On Thu, 2023-10-26 at 12:33 +0200, Alexander Kanavin wrote: > > 'bitbake -S printdiff' is a useful diagnostic facility for finding out > > why sstate is not being reused, but until now it had no tests that would > > ensure it works. This commit adds three basic scenarios: > > > > 1. make a change in a really basic, common recipe that is at the very root > > of dependency trees (quilt-native), and ensure that change is correctly > > discovered when > > building an image. > > > > 2. make a change in gcc-source recipe, which is somewhat special > > (operates in work-shared), and ensure that gcc-runtime builds track > > that down as well. > > > > 3. make a change in base_do_configure() definition from base.bbclass, > > which is not recipe-specific, but affects many basic recipes, and ensure > > that > > is correctly reported as well. > > > > The test itself actually runs twice: > > - first against a fully populated build directory, where > > the printdiff code is guaranteed to find the correct previous > > stamp that can be compared with in a predictable manner. > > > > - then in an empty build directory where the printdiff code > > goes to look in the sstate cache, and so the existence of the > > previous signature can be tested, but not the difference with it > > (what the exact difference would be is unpredictable as the > > sstate cache is indeed shared between many builds). > > > > Signed-off-by: Alexander Kanavin <a...@linutronix.de>
There have been autobulilder failures from the do_configure base change and we've not been quite sure why. The summary below took quite a bit of digging but seems quite simple when written up like this! If you dig into a failure like: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2379/steps/15/logs/stdio you see messages like: The differences between the current build and any cached tasks start at the following tasks: /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_collect_spdx_deps virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/zlib/zlib_1.3.bb:do_collect_spdx_deps virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_collect_spdx_deps /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot which seems odd. Taking the first one, if quilt-native has built, you'd expect those to be present. I went onto the arm builder and tried a "bitbake quilt-native -S printdiff -DDD": DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/4d/8d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:4d8da10af56c40d61adb5febb96f583a399ceb276d0dc13e61971f1eb0b7d81d_create_runtime_spdx.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/96/sstate:quilt-native::0.67:r0::11:7396e70fb1865087409e454eb8bacae3bcb4add6218b3cd707d2e96880b72936_populate_lic.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/1b/da/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:1bdafcc9e6b4209bfbdf4b83f8b44e0cc3cc26cdeebe7d713a9f7fcba4b02d48_create_spdx.tar.zst.siginfo and the sstate for quilt-native_0.67.bb:do_collect_spdx_deps is indeed missing, there is only x86: $ ls /srv/autobuilder/autobuilder.yocto.io/pub/sstate/*/72/12/sstate\:quilt-native\:*collect_spdx_deps.tar.zst.siginfo /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:x86_64-linux:0.67:r0:x86_64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo The normal build works since it doesn't care about the siginfo files, just the actual sstate it needs and as a covered task, it just skips over this. This highlights two issues: a) Why is the single sstate siginfo file gone? The most likely explanation is that it wasn't accessed recently and was deleted by the cleanup/ageing code. This makes sense since the code in question never downloads it, only checks it exists and uses that as a marker. Our usage "accesses" are in a different codepath to the parsing check. b) This means -S printdiff doesn't quite do what you'd expect since it doesn't skip "covered" tasks in the same way a normal build would. To reproduce the test failure you should be able to just delete that siginfo file from your local sstate. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#190209): https://lists.openembedded.org/g/openembedded-core/message/190209 Mute This Topic: https://lists.openembedded.org/mt/102197154/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
