Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
Thank you both for your comments. Jose, I just sent a fix with --newbuildir, it's quite convenient in this case, thx. Fabien -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61336): https://lists.yoctoproject.org/g/yocto/message/61336 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
Alexander Kanavin escreveu no dia quarta, 11/10/2023 à(s) 09:38: > On Fri, 6 Oct 2023 at 16:27, Fabien Thomas wrote: > > -> On the build step "Run cmds" for meta-oe, I see following error : > > ERROR - Build directory > > /home/pokybuild/yocto-worker/reproducible-meta-oe/build/build-st > > already exists, aborting > > Probably due to previous step failed. Do you think it is necessary to > > add an rm -rf [...]/build-st in each step, before actually running > > selftest ? > > No, because this complicates investigating failures after they > happened. The correct thing to do is to stop the whole job when a > specific step fails. Letting subsequent steps fail due to the already > existing failed directory is ok too. > Other options can be using oe-selftest --newbuilddir [...]/build-st-${LAYER} and build each layer in your own directory. I have used this --newbuilddir in my reproducible build and it works. Jose > > Alex > > > > -- Best regards, José Quaresma -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61320): https://lists.yoctoproject.org/g/yocto/message/61320 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
On Fri, 6 Oct 2023 at 16:27, Fabien Thomas wrote: > -> On the build step "Run cmds" for meta-oe, I see following error : > ERROR - Build directory > /home/pokybuild/yocto-worker/reproducible-meta-oe/build/build-st > already exists, aborting > Probably due to previous step failed. Do you think it is necessary to > add an rm -rf [...]/build-st in each step, before actually running > selftest ? No, because this complicates investigating failures after they happened. The correct thing to do is to stop the whole job when a specific step fails. Letting subsequent steps fail due to the already existing failed directory is ok too. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61316): https://lists.yoctoproject.org/g/yocto/message/61316 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
Thank you Richard for trying this patch, this report is very helpful. -> Well noted for the short name, indeed it's absolutely not readable in the UI, I will go with this name "Repro test OE meta-xxx" if you agree. -> Reading the logs, I see that all "Add layers" steps always fail except meta-initramfs and meta-oe, both has no dependencies. I understand how it works now ! I had to respect the dependencies order in the "ADDLAYER" list because it uses the bitbake-layers add-layer cmd. It will be fixed too. -> On the build step "Run cmds" for meta-oe, I see following error : ERROR - Build directory /home/pokybuild/yocto-worker/reproducible-meta-oe/build/build-st already exists, aborting Probably due to previous step failed. Do you think it is necessary to add an rm -rf [...]/build-st in each step, before actually running selftest ? -> Also, I would like to inform you that I'm having build problems with every layer in the reproducible sefltest context, except with meta-initramfs. So I focused myself on meta-oe. One of the building issues is regarding recipes that use the useradd class. In the reproducible context, these must use static-id. I fixed it by creating new files/static-passwd and files/static-group into meta-oe layer. I plan to send patches to both yocto-autobuilder-helper & meta-openmebdded project soon. Also I had trouble with 4 recipes (webkitgtk3_2.40.5, influxdb_1.8.10, xorgxrdp_0.9.19 and xscreensaver_6.04). I didn't investigate but only excluded them from the world target. So my patch might look like this : "step6" : { "shortname" : "Repro test OE meta-oe", "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded/meta-oe/ DISPLAY=:1 oe-selftest -r reproducible"], "ADDLAYER" : [ "${BUILDDIR}/../meta-openembedded/meta-oe" ], "extravars" : [ "USERADD_UID_TABLES:append = ' files/static-passwd-meta-oe'", "USERADD_GID_TABLES:append = ' files/static-group-meta-oe'" "EXCLUDE_FROM_WORLD:pn-xscreensaver" "EXCLUDE_FROM_WORLD:pn-webkitgtk3" "EXCLUDE_FROM_WORLD:pn-influxdb" "EXCLUDE_FROM_WORLD:pn-xorgxrdp" ] }, What do you think ? (Note that for layers with dependencies all static-* files should be added) -> With these adjustments, I finally had a successful build on my computer. FYI here are the results for meta-oe : oe-selftest - INFO - Reproducibility summary for deb: same=14650 different=128 different_excluded=0 missing=0 total=14778 Le jeu. 5 oct. 2023 à 23:51, Richard Purdie a écrit : > > On Thu, 2023-10-05 at 20:46 +0100, Richard Purdie via > lists.yoctoproject.org wrote: > > On Thu, 2023-10-05 at 17:42 +0100, Richard Purdie via > > lists.yoctoproject.org wrote: > > > On Mon, 2023-10-02 at 16:54 +0200, Fabien Thomas wrote: > > > > The purpose of this new builder is to report the reproducibility status > > > > of all meta-openembedded recipes layer by layer. It use the same > > > > reproducible selftest than OE-Core but setting only world as target, > > > > and excluding all oecore and other openembedded layer recipes. > > > > Also, the report output directory is split by layers. > > > > > > > > Signed-off-by: Fabien Thomas > > > > Reviewed-by: Yoann Congal > > > > --- > > > > > > > > Changes v1->v2 : > > > > * Split builder into multiple steps, one for each openembedded layers. > > > > * Split reports output in the same way, one for each layers. > > > > * For each step, only to be tested layer and its dependancies are added. > > > > * Every other layers than the one that is tested is excluded from world. > > > > > > Thanks, I've merged this with a tweak of "reproducible-meta- > > > openembedded" -> "reproducible-meta-oe" just to keep things more > > > readable in the UI. > > > > > > There is a test run queued here: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/buildrequests/454516?redirect_to_build=true > > > > > > but things are a bit busy atm on the autobuilder so it is queued. > > > > That broke as did the second attempt. I've pushed controller side > > tweaks and some tweaks to helper which means this is now running: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/155/builds/3 > > > > Obviously there are still some failures but it is doing more this time > > and a least running something in some steps. > > Whilst a lot of it failed, there was one piece that succeeded and > generated diffoscope output: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/155/builds/3/steps/21/logs/stdio > http://autobuilder.yocto.io/pub/repro-fail-openembedded-meta-initramfs/oe-reproducible-20231005-y6zo9rio/packages/diff-html/ > > As well as fixing the failures, the "short" names need work to make the > UI readable. > > Cheers, > > Richard
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
On Thu, 2023-10-05 at 20:46 +0100, Richard Purdie via lists.yoctoproject.org wrote: > On Thu, 2023-10-05 at 17:42 +0100, Richard Purdie via > lists.yoctoproject.org wrote: > > On Mon, 2023-10-02 at 16:54 +0200, Fabien Thomas wrote: > > > The purpose of this new builder is to report the reproducibility status > > > of all meta-openembedded recipes layer by layer. It use the same > > > reproducible selftest than OE-Core but setting only world as target, > > > and excluding all oecore and other openembedded layer recipes. > > > Also, the report output directory is split by layers. > > > > > > Signed-off-by: Fabien Thomas > > > Reviewed-by: Yoann Congal > > > --- > > > > > > Changes v1->v2 : > > > * Split builder into multiple steps, one for each openembedded layers. > > > * Split reports output in the same way, one for each layers. > > > * For each step, only to be tested layer and its dependancies are added. > > > * Every other layers than the one that is tested is excluded from world. > > > > Thanks, I've merged this with a tweak of "reproducible-meta- > > openembedded" -> "reproducible-meta-oe" just to keep things more > > readable in the UI. > > > > There is a test run queued here: > > > > https://autobuilder.yoctoproject.org/typhoon/#/buildrequests/454516?redirect_to_build=true > > > > but things are a bit busy atm on the autobuilder so it is queued. > > That broke as did the second attempt. I've pushed controller side > tweaks and some tweaks to helper which means this is now running: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/155/builds/3 > > Obviously there are still some failures but it is doing more this time > and a least running something in some steps. Whilst a lot of it failed, there was one piece that succeeded and generated diffoscope output: https://autobuilder.yoctoproject.org/typhoon/#/builders/155/builds/3/steps/21/logs/stdio http://autobuilder.yocto.io/pub/repro-fail-openembedded-meta-initramfs/oe-reproducible-20231005-y6zo9rio/packages/diff-html/ As well as fixing the failures, the "short" names need work to make the UI readable. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61251): https://lists.yoctoproject.org/g/yocto/message/61251 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
On Thu, 2023-10-05 at 17:42 +0100, Richard Purdie via lists.yoctoproject.org wrote: > On Mon, 2023-10-02 at 16:54 +0200, Fabien Thomas wrote: > > The purpose of this new builder is to report the reproducibility status > > of all meta-openembedded recipes layer by layer. It use the same > > reproducible selftest than OE-Core but setting only world as target, > > and excluding all oecore and other openembedded layer recipes. > > Also, the report output directory is split by layers. > > > > Signed-off-by: Fabien Thomas > > Reviewed-by: Yoann Congal > > --- > > > > Changes v1->v2 : > > * Split builder into multiple steps, one for each openembedded layers. > > * Split reports output in the same way, one for each layers. > > * For each step, only to be tested layer and its dependancies are added. > > * Every other layers than the one that is tested is excluded from world. > > Thanks, I've merged this with a tweak of "reproducible-meta- > openembedded" -> "reproducible-meta-oe" just to keep things more > readable in the UI. > > There is a test run queued here: > > https://autobuilder.yoctoproject.org/typhoon/#/buildrequests/454516?redirect_to_build=true > > but things are a bit busy atm on the autobuilder so it is queued. That broke as did the second attempt. I've pushed controller side tweaks and some tweaks to helper which means this is now running: https://autobuilder.yoctoproject.org/typhoon/#/builders/155/builds/3 Obviously there are still some failures but it is doing more this time and a least running something in some steps. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61250): https://lists.yoctoproject.org/g/yocto/message/61250 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
On Mon, 2023-10-02 at 16:54 +0200, Fabien Thomas wrote: > The purpose of this new builder is to report the reproducibility status > of all meta-openembedded recipes layer by layer. It use the same > reproducible selftest than OE-Core but setting only world as target, > and excluding all oecore and other openembedded layer recipes. > Also, the report output directory is split by layers. > > Signed-off-by: Fabien Thomas > Reviewed-by: Yoann Congal > --- > > Changes v1->v2 : > * Split builder into multiple steps, one for each openembedded layers. > * Split reports output in the same way, one for each layers. > * For each step, only to be tested layer and its dependancies are added. > * Every other layers than the one that is tested is excluded from world. Thanks, I've merged this with a tweak of "reproducible-meta- openembedded" -> "reproducible-meta-oe" just to keep things more readable in the UI. There is a test run queued here: https://autobuilder.yoctoproject.org/typhoon/#/buildrequests/454516?redirect_to_build=true but things are a bit busy atm on the autobuilder so it is queued. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61248): https://lists.yoctoproject.org/g/yocto/message/61248 Mute This Topic: https://lists.yoctoproject.org/mt/101713575/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
looks fine to me Thanks On Mon, Oct 2, 2023 at 7:56 AM Fabien Thomas wrote: > > The purpose of this new builder is to report the reproducibility status > of all meta-openembedded recipes layer by layer. It use the same > reproducible selftest than OE-Core but setting only world as target, > and excluding all oecore and other openembedded layer recipes. > Also, the report output directory is split by layers. > > Signed-off-by: Fabien Thomas > Reviewed-by: Yoann Congal > --- > > Changes v1->v2 : > * Split builder into multiple steps, one for each openembedded layers. > * Split reports output in the same way, one for each layers. > * For each step, only to be tested layer and its dependancies are added. > * Every other layers than the one that is tested is excluded from world. > > config.json | 133 > 1 file changed, 133 insertions(+) > > diff --git a/config.json b/config.json > index 05c6794..90762f9 100644 > --- a/config.json > +++ b/config.json > @@ -264,6 +264,136 @@ > > } > }, > +"reproducible-meta-openembedded" : { > +"MACHINE" : "qemux86-64", > +"SDKMACHINE" : "x86_64", > +"DISTRO" : "None", > +"NEEDREPOS" : ["oecore", "bitbake", "meta-openembedded"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-selftest" > +], > +"extravars" : [ > +"EXCLUDE_FROM_WORLD:layer-core = '1'", > +"EXCLUDE_FROM_WORLD:layer-selftest = '1'", > +"OEQA_REPRODUCIBLE_TEST_TARGET = 'world'" > +], > +"step1" : { > +"shortname" : "Reproducible Selftest for openembedded > meta-filesystems layer", > +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-filesystems/ > DISPLAY=:1 oe-selftest -r reproducible"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-openembedded/meta-filesystems", > +"${BUILDDIR}/../meta-openembedded/meta-oe" > +], > +"extravars" : [ > +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'" > +] > +}, > +"step2" : { > +"shortname" : "Reproducible Selftest for openembedded > meta-gnome layer", > +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-gnome/ > DISPLAY=:1 oe-selftest -r reproducible"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-openembedded/meta-gnome", > +"${BUILDDIR}/../meta-openembedded/meta-oe", > +"${BUILDDIR}/../meta-openembedded/meta-networking", > +"${BUILDDIR}/../meta-openembedded/meta-python" > +], > +"extravars" : [ > +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'", > +"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'", > +"EXCLUDE_FROM_WORLD:layer-meta-python = '1'" > +] > +}, > +"step3" : { > +"shortname" : "Reproducible Selftest for openembedded > meta-initramfs layer", > +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-initramfs/ > DISPLAY=:1 oe-selftest -r reproducible"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-openembedded/meta-initramfs" > +] > +}, > +"step4" : { > +"shortname" : "Reproducible Selftest for openembedded > meta-multimedia layer", > +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-multimedia/ > DISPLAY=:1 oe-selftest -r reproducible"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-openembedded/meta-multimedia", > +"${BUILDDIR}/../meta-openembedded/meta-oe", > +"${BUILDDIR}/../meta-openembedded/meta-python" > +], > +"extravars" : [ > +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'", > +"EXCLUDE_FROM_WORLD:layer-meta-python = '1'" > +] > +}, > +"step5" : { > +"shortname" : "Reproducible Selftest for openembedded > meta-networking layer", > +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; > OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-networking/ > DISPLAY=:1 oe-selftest -r reproducible"], > +"ADDLAYER" : [ > +"${BUILDDIR}/../meta-openembedded/meta-networking", > +
[yocto] [yocto-autobuilder-helper][PATCH v2] config.json: add reproducible-openembedded build
The purpose of this new builder is to report the reproducibility status of all meta-openembedded recipes layer by layer. It use the same reproducible selftest than OE-Core but setting only world as target, and excluding all oecore and other openembedded layer recipes. Also, the report output directory is split by layers. Signed-off-by: Fabien Thomas Reviewed-by: Yoann Congal --- Changes v1->v2 : * Split builder into multiple steps, one for each openembedded layers. * Split reports output in the same way, one for each layers. * For each step, only to be tested layer and its dependancies are added. * Every other layers than the one that is tested is excluded from world. config.json | 133 1 file changed, 133 insertions(+) diff --git a/config.json b/config.json index 05c6794..90762f9 100644 --- a/config.json +++ b/config.json @@ -264,6 +264,136 @@ } }, +"reproducible-meta-openembedded" : { +"MACHINE" : "qemux86-64", +"SDKMACHINE" : "x86_64", +"DISTRO" : "None", +"NEEDREPOS" : ["oecore", "bitbake", "meta-openembedded"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-selftest" +], +"extravars" : [ +"EXCLUDE_FROM_WORLD:layer-core = '1'", +"EXCLUDE_FROM_WORLD:layer-selftest = '1'", +"OEQA_REPRODUCIBLE_TEST_TARGET = 'world'" +], +"step1" : { +"shortname" : "Reproducible Selftest for openembedded meta-filesystems layer", +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-filesystems/ DISPLAY=:1 oe-selftest -r reproducible"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-openembedded/meta-filesystems", +"${BUILDDIR}/../meta-openembedded/meta-oe" +], +"extravars" : [ +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'" +] +}, +"step2" : { +"shortname" : "Reproducible Selftest for openembedded meta-gnome layer", +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-gnome/ DISPLAY=:1 oe-selftest -r reproducible"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-openembedded/meta-gnome", +"${BUILDDIR}/../meta-openembedded/meta-oe", +"${BUILDDIR}/../meta-openembedded/meta-networking", +"${BUILDDIR}/../meta-openembedded/meta-python" +], +"extravars" : [ +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'", +"EXCLUDE_FROM_WORLD:layer-networking-layer = '1'", +"EXCLUDE_FROM_WORLD:layer-meta-python = '1'" +] +}, +"step3" : { +"shortname" : "Reproducible Selftest for openembedded meta-initramfs layer", +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-initramfs/ DISPLAY=:1 oe-selftest -r reproducible"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-openembedded/meta-initramfs" +] +}, +"step4" : { +"shortname" : "Reproducible Selftest for openembedded meta-multimedia layer", +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-multimedia/ DISPLAY=:1 oe-selftest -r reproducible"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-openembedded/meta-multimedia", +"${BUILDDIR}/../meta-openembedded/meta-oe", +"${BUILDDIR}/../meta-openembedded/meta-python" +], +"extravars" : [ +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'", +"EXCLUDE_FROM_WORLD:layer-meta-python = '1'" +] +}, +"step5" : { +"shortname" : "Reproducible Selftest for openembedded meta-networking layer", +"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail-openembedded-meta-networking/ DISPLAY=:1 oe-selftest -r reproducible"], +"ADDLAYER" : [ +"${BUILDDIR}/../meta-openembedded/meta-networking", +"${BUILDDIR}/../meta-openembedded/meta-oe" +], +"extravars" : [ +"EXCLUDE_FROM_WORLD:layer-openembedded-layer = '1'" +] +}, +"step6" : { +"shortname" : "Reproducible Selftest for