Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
Thank you both for your comments. Jose, I just sent a fix with --newbuildir, it's quite convenient in this case, thx. Fabien -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61336): https://lists.yoctoproject.org/g/yocto/message/61336 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[yocto] [yocto-autobuilder-helper][PATCH] config.json : Fix repro-meta-oe layer dependency
Fix meta-filesystem build issue due to hidden dependencies (not set by
LAYERDEPENDS) smbnetfs(meta-filesystem) depends on
samba(meta-networking) that depends on python3-dnspython (meta-python)
Create new build-st dir for each layer to avoid chain of
failure between steps
Signed-off-by: Fabien Thomas
Reviewed-by: Yoann Congal
---
config.json | 22 --
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/config.json b/config.json
index 3acb710..c01a453 100644
--- a/config.json
+++ b/config.json
@@ -303,15 +303,17 @@
],
"step1" : {
"shortname" : "Repro meta-oe/meta-filesystems",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-filesystems/
DISPLAY=:1 oe-selftest -r reproducible"],
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-filesystems/
DISPLAY=:1 oe-selftest --newbuilddir $BBPATH/build-st-meta-filesystems -r
reproducible"],
"ADDLAYER" : [
"${BUILDDIR}/../meta-openembedded/meta-oe",
+"${BUILDDIR}/../meta-openembedded/meta-python",
"${BUILDDIR}/../meta-openembedded/meta-networking",
"${BUILDDIR}/../meta-openembedded/meta-filesystems"
],
"extravars" : [
"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'",
"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'",
+"EXCLUDE_FROM_WORLD:layer-meta-python = '1'",
"USERADD_UID_TABLES:append = '
files/static-passwd-meta-filesystems'",
"USERADD_GID_TABLES:append = '
files/static-group-meta-networking'",
"USERADD_UID_TABLES:append = '
files/static-passwd-meta-networking'",
@@ -321,7 +323,7 @@
},
"step2" : {
"shortname" : "Repro meta-oe/meta-gnome",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-gnome/
DISPLAY=:1 oe-selftest -r reproducible"],
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-gnome/
DISPLAY=:1 oe-selftest --newbuilddir $BBPATH/build-st-meta-gnome -r
reproducible"],
"ADDLAYER" : [
"${BUILDDIR}/../meta-openembedded/meta-oe",
"${BUILDDIR}/../meta-openembedded/meta-python",
@@ -341,14 +343,14 @@
},
"step3" : {
"shortname" : "Repro meta-oe/meta-initramfs",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-initramfs/
DISPLAY=:1 oe-selftest -r reproducible"],
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-initramfs/
DISPLAY=:1 oe-selftest --newbuilddir $BBPATH/build-st-meta-initramfs -r
reproducible"],
"ADDLAYER" : [
"${BUILDDIR}/../meta-openembedded/meta-initramfs"
]
},
"step4" : {
"shortname" : "Repro meta-oe/meta-multimedia",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-multimedia/
DISPLAY=:1 oe-selftest -r reproducible"],
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-multimedia/
DISPLAY=:1 oe-selftest --newbuilddir $BBPATH/build-st-meta-multimedia -r
reproducible"],
"ADDLAYER" : [
"${BUILDDIR}/../meta-openembedded/meta-oe",
"${BUILDDIR}/../meta-openembedded/meta-python",
@@ -365,7 +367,7 @@
},
"step5" : {
"shortname" : "Repro meta-oe/meta-networking",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-networking/
DISPLAY=:1 oe-selftest -r reproducible"],
+"EXTRACMDS" : ["${SCRI
[yocto] [yocto-autobuilder-helper][PATCH] config.json : reproducible-meta-oe build fixes
- Distro has been changed to Poky so exclude Poky's layers from world to
limit reproducibility tests to the targeted layer only.
- Execution on autobuilder shows that addlayer steps often fail :
reorder layer in ADDLAYER list regarding their dependencies.
- Fix shortid building issues by setting USERADD_U(G)ID_TABLES to
passwd and group files. These files must exist in meta-openmbedded.
- Shorten the steps shortnames to improve readability on UI.
- Add a directory level for output reports, for convenience.
Signed-off-by: Fabien Thomas
Reviewed-by: Yoann Congal
---
config.json | 107 ++--
1 file changed, 70 insertions(+), 37 deletions(-)
diff --git a/config.json b/config.json
index df3493b..af4ab39 100644
--- a/config.json
+++ b/config.json
@@ -296,116 +296,149 @@
],
"extravars" : [
"EXCLUDE_FROM_WORLD:layer-core = '1'",
+"EXCLUDE_FROM_WORLD:layer-yocto = '1'",
+"EXCLUDE_FROM_WORLD:layer-yoctobsp = '1'",
"EXCLUDE_FROM_WORLD:layer-selftest = '1'",
"OEQA_REPRODUCIBLE_TEST_TARGET = 'world'"
],
"step1" : {
-"shortname" : "Reproducible Selftest for openembedded
meta-filesystems layer",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-filesystems/
DISPLAY=:1 oe-selftest -r reproducible"],
+"shortname" : "Repro test OE meta-filesystems",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-filesystems/
DISPLAY=:1 oe-selftest -r reproducible"],
"ADDLAYER" : [
-"${BUILDDIR}/../meta-openembedded/meta-filesystems",
"${BUILDDIR}/../meta-openembedded/meta-oe"
+"${BUILDDIR}/../meta-openembedded/meta-networking",
+"${BUILDDIR}/../meta-openembedded/meta-filesystems",
],
"extravars" : [
-"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'"
+"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'",
+"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'",
+"USERADD_UID_TABLES:append = '
files/static-passwd-meta-filesystems'",
+"USERADD_GID_TABLES:append = '
files/static-group-meta-networking'",
+"USERADD_UID_TABLES:append = '
files/static-passwd-meta-networking'",
+"USERADD_GID_TABLES:append = '
files/static-group-meta-oe'",
+"USERADD_UID_TABLES:append = '
files/static-passwd-meta-oe'"
]
},
"step2" : {
-"shortname" : "Reproducible Selftest for openembedded
meta-gnome layer",
-"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-gnome/
DISPLAY=:1 oe-selftest -r reproducible"],
+"shortname" : "Repro test OE meta-gnome",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-gnome/
DISPLAY=:1 oe-selftest -r reproducible"],
"ADDLAYER" : [
-"${BUILDDIR}/../meta-openembedded/meta-gnome",
"${BUILDDIR}/../meta-openembedded/meta-oe",
-"${BUILDDIR}/../meta-openembedded/meta-networking",
"${BUILDDIR}/../meta-openembedded/meta-python"
+"${BUILDDIR}/../meta-openembedded/meta-networking",
+"${BUILDDIR}/../meta-openembedded/meta-gnome",
],
"extravars" : [
"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'",
"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'",
-"EXCLUDE_FROM_WORLD:layer-meta-python = '1'"
+"EXCLUDE_FROM_WORLD:layer-meta-python = '1'",
+"USERADD_UID_TABLES:append = '
files/static-passwd-meta-gnome'",
+"USERADD_GID_TABLES:append = '
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
Thank you Richard for trying this patch, this report is very helpful.
-> Well noted for the short name, indeed it's absolutely not readable
in the UI, I will go with this name "Repro test OE meta-xxx" if you
agree.
-> Reading the logs, I see that all "Add layers" steps always fail
except meta-initramfs and meta-oe, both has no dependencies.
I understand how it works now ! I had to respect the dependencies
order in the "ADDLAYER" list because it uses the bitbake-layers
add-layer cmd.
It will be fixed too.
-> On the build step "Run cmds" for meta-oe, I see following error :
ERROR - Build directory
/home/pokybuild/yocto-worker/reproducible-meta-oe/build/build-st
already exists, aborting
Probably due to previous step failed. Do you think it is necessary to
add an rm -rf [...]/build-st in each step, before actually running
selftest ?
-> Also, I would like to inform you that I'm having build problems
with every layer in the reproducible sefltest context, except with
meta-initramfs.
So I focused myself on meta-oe.
One of the building issues is regarding recipes that use the useradd
class. In the reproducible context, these must use static-id.
I fixed it by creating new files/static-passwd and files/static-group
into meta-oe layer. I plan to send patches to both
yocto-autobuilder-helper & meta-openmebdded project soon.
Also I had trouble with 4 recipes (webkitgtk3_2.40.5, influxdb_1.8.10,
xorgxrdp_0.9.19 and xscreensaver_6.04). I didn't investigate but only
excluded them from the world target.
So my patch might look like this :
"step6" : {
"shortname" : "Repro test OE meta-oe",
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-oe/
DISPLAY=:1 oe-selftest -r reproducible"],
"ADDLAYER" : [
"${BUILDDIR}/../meta-openembedded/meta-oe"
],
"extravars" : [
"USERADD_UID_TABLES:append = '
files/static-passwd-meta-oe'",
"USERADD_GID_TABLES:append = ' files/static-group-meta-oe'"
"EXCLUDE_FROM_WORLD:pn-xscreensaver"
"EXCLUDE_FROM_WORLD:pn-webkitgtk3"
"EXCLUDE_FROM_WORLD:pn-influxdb"
"EXCLUDE_FROM_WORLD:pn-xorgxrdp"
]
},
What do you think ? (Note that for layers with dependencies all
static-* files should be added)
-> With these adjustments, I finally had a successful build on my
computer. FYI here are the results for meta-oe :
oe-selftest - INFO - Reproducibility summary for deb: same=14650
different=128 different_excluded=0 missing=0 total=14778
Le jeu. 5 oct. 2023 à 23:51, Richard Purdie
a écrit :
>
> On Thu, 2023-10-05 at 20:46 +0100, Richard Purdie via
> lists.yoctoproject.org wrote:
> > On Thu, 2023-10-05 at 17:42 +0100, Richard Purdie via
> > lists.yoctoproject.org wrote:
> > > On Mon, 2023-10-02 at 16:54 +0200, Fabien Thomas wrote:
> > > > The purpose of this new builder is to report the reproducibility status
> > > > of all meta-openembedded recipes layer by layer. It use the same
> > > > reproducible selftest than OE-Core but setting only world as target,
> > > > and excluding all oecore and other openembedded layer recipes.
> > > > Also, the report output directory is split by layers.
> > > >
> > > > Signed-off-by: Fabien Thomas
> > > > Reviewed-by: Yoann Congal
> > > > ---
> > > >
> > > > Changes v1->v2 :
> > > > * Split builder into multiple steps, one for each openembedded layers.
> > > > * Split reports output in the same way, one for each layers.
> > > > * For each step, only to be tested layer and its dependancies are added.
> > > > * Every other layers than the one that is tested is excluded from world.
> > >
> > > Thanks, I've merged this with a tweak of "reproducible-meta-
> > > openembedded" -> "reproducible-meta-oe" just to keep things more
> > > readable in the UI.
> > >
> > > There is a test run queued here:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/buildrequests/454516?redirect_to_build=true
> > >
> > > but things are a bit busy atm on the autobuilder so it is queued.
> >
> > That broke as did the second attempt. I've pushed controller side
> > tweaks and some tweaks to helper which means this is now running:
> >
> > https://autobuilder.yoctoproject
[yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
The purpose of this new builder is to report the reproducibility status
of all meta-openembedded recipes layer by layer. It use the same
reproducible selftest than OE-Core but setting only world as target,
and excluding all oecore and other openembedded layer recipes.
Also, the report output directory is split by layers.
Signed-off-by: Fabien Thomas
Reviewed-by: Yoann Congal
---
Changes v1->v2 :
* Split builder into multiple steps, one for each openembedded layers.
* Split reports output in the same way, one for each layers.
* For each step, only to be tested layer and its dependancies are added.
* Every other layers than the one that is tested is excluded from world.
config.json | 133
1 file changed, 133 insertions(+)
diff --git a/config.json b/config.json
index 05c6794..90762f9 100644
--- a/config.json
+++ b/config.json
@@ -264,6 +264,136 @@
}
},
+"reproducible-meta-openembedded" : {
+"MACHINE" : "qemux86-64",
+"SDKMACHINE" : "x86_64",
+"DISTRO" : "None",
+"NEEDREPOS" : ["oecore", "bitbake", "meta-openembedded"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-selftest"
+],
+"extravars" : [
+"EXCLUDE_FROM_WORLD:layer-core = '1'",
+"EXCLUDE_FROM_WORLD:layer-selftest = '1'",
+"OEQA_REPRODUCIBLE_TEST_TARGET = 'world'"
+],
+"step1" : {
+"shortname" : "Reproducible Selftest for openembedded
meta-filesystems layer",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-filesystems/
DISPLAY=:1 oe-selftest -r reproducible"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-openembedded/meta-filesystems",
+"${BUILDDIR}/../meta-openembedded/meta-oe"
+],
+"extravars" : [
+"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'"
+]
+},
+"step2" : {
+"shortname" : "Reproducible Selftest for openembedded
meta-gnome layer",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-gnome/
DISPLAY=:1 oe-selftest -r reproducible"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-openembedded/meta-gnome",
+"${BUILDDIR}/../meta-openembedded/meta-oe",
+"${BUILDDIR}/../meta-openembedded/meta-networking",
+"${BUILDDIR}/../meta-openembedded/meta-python"
+],
+"extravars" : [
+"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'",
+"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'",
+"EXCLUDE_FROM_WORLD:layer-meta-python = '1'"
+]
+},
+"step3" : {
+"shortname" : "Reproducible Selftest for openembedded
meta-initramfs layer",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-initramfs/
DISPLAY=:1 oe-selftest -r reproducible"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-openembedded/meta-initramfs"
+]
+},
+"step4" : {
+"shortname" : "Reproducible Selftest for openembedded
meta-multimedia layer",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-multimedia/
DISPLAY=:1 oe-selftest -r reproducible"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-openembedded/meta-multimedia",
+"${BUILDDIR}/../meta-openembedded/meta-oe",
+"${BUILDDIR}/../meta-openembedded/meta-python"
+],
+"extravars" : [
+"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'",
+"EXCLUDE_FROM_WORLD:layer-meta-python = '1'"
+]
+},
+"step5" :
[yocto] [PATCH][yocto-autobuilder-helper] config.json: add reproducible-openembedded build
The purpose of this new build is to report the reproducibility status of
all meta-openembedded recipes. It use the same reproducible selftest than
OE-Core but setting only world as target, and excluding all oecore recipes
from it. Also, the report output directory for meta-openembedded is
separate from oecore's one.
Signed-off-by: Fabien Thomas
Reviewed-by: Yoann Congal
---
config.json | 31 +++
1 file changed, 31 insertions(+)
diff --git a/config.json b/config.json
index 05c6794..fde1b1f 100644
--- a/config.json
+++ b/config.json
@@ -264,6 +264,34 @@
}
},
+"reproducible-openembedded" : {
+"MACHINE" : "qemux86-64",
+"SDKMACHINE" : "x86_64",
+"DISTRO" : "None",
+"NEEDREPOS" : ["oecore", "bitbake", "meta-openembedded"],
+"extravars" : [
+"EXCLUDE_FROM_WORLD:layer-core = '1'",
+"EXCLUDE_FROM_WORLD:layer-selftest = '1'",
+"OEQA_REPRODUCIBLE_TEST_TARGET = 'world'"
+],
+"step1" : {
+"shortname" : "Reproducible Selftest for openembedded layers",
+"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc;
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/
DISPLAY=:1 oe-selftest -r reproducible"],
+"ADDLAYER" : [
+"${BUILDDIR}/../meta-selftest",
+"${BUILDDIR}/../meta-openembedded/meta-filesystems",
+"${BUILDDIR}/../meta-openembedded/meta-gnome",
+"${BUILDDIR}/../meta-openembedded/meta-initramfs",
+"${BUILDDIR}/../meta-openembedded/meta-multimedia",
+"${BUILDDIR}/../meta-openembedded/meta-networking",
+"${BUILDDIR}/../meta-openembedded/meta-oe",
+"${BUILDDIR}/../meta-openembedded/meta-perl",
+"${BUILDDIR}/../meta-openembedded/meta-python",
+"${BUILDDIR}/../meta-openembedded/meta-webserver",
+"${BUILDDIR}/../meta-openembedded/meta-xfce"
+]
+}
+},
"trigger-build" : {
"SDKMACHINE" : "x86_64",
"MACHINE" : "qemux86-64",
@@ -955,6 +983,9 @@
"reproducible" : {
"TEMPLATE" : "reproducible"
},
+"reproducible-openembedded" : {
+"TEMPLATE" : "reproducible-openembedded"
+},
"reproducible-ubuntu" : {
"TEMPLATE" : "reproducible"
},
--
2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61153): https://lists.yoctoproject.org/g/yocto/message/61153
Mute This Topic: https://lists.yoctoproject.org/mt/101637262/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
