Thanks for working on this! I finally got to playing with CDN mirror a bit, and it seems to basically work ok, so next I'm going to write AB tests that check that it remains useful. Specifically:
1. What should the test be? I tried 'bitbake -S printdiff core-image-sato' on a blank build directory, and the result against current master is: =========== Checking sstate mirror object availability: 100% |##################################################################################################################################################| Time: 0:05:35 The differences between the current build and any cached tasks start at the following tasks: /srv/work/alex/poky/meta/recipes-sato/images/core-image-sato.bb:do_deploy_source_date_epoch /srv/work/alex/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_install =========== I think this is pretty good! The two missing objects depend on the current date, and so should go to the exception list in the test, and otherwise there is a 100% match rate. The bitbake targets could be 'world core-image-sato-sdk', and target machines could be qemux86_64 and qemuarm64. Just to be sure that mismatches elsewhere will be reported as CDN misses, adding my shadow 4.14 update and re-running the command shows: ============ The differences between the current build and any cached tasks start at the following tasks: /srv/work/alex/poky/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.17.bb:do_package /srv/work/alex/poky/meta/recipes-sato/pulseaudio-sato/pulseaudio-client-conf-sato_1.bb:do_package /srv/work/alex/poky/meta/recipes-core/initscripts/initscripts_1.0.bb:do_package /srv/work/alex/poky/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_package /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-ucm-conf_1.2.10.bb:do_package /srv/work/alex/poky/meta/recipes-kernel/wireless-regdb/wireless-regdb_2023.09.01.bb:do_package /srv/work/alex/poky/meta/recipes-sato/packagegroups/packagegroup-core-x11-sato.bb:do_package /srv/work/alex/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_6.5.bb:do_package /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-ssh-dropbear.bb:do_package /srv/work/alex/poky/meta/recipes-graphics/packagegroups/packagegroup-core-x11.bb:do_package /srv/work/alex/poky/meta/recipes-sato/shutdown-desktop/shutdown-desktop.bb:do_package virtual:native:/srv/work/alex/poky/meta/recipes-support/libbsd/libbsd_0.11.7.bb:do_prepare_recipe_sysroot /srv/work/alex/poky/meta/recipes-support/ca-certificates/ca-certificates_20211016.bb:do_package /srv/work/alex/poky/meta/recipes-core/sysvinit/sysvinit-inittab_2.88dsf.bb:do_package virtual:native:/srv/work/alex/poky/meta/recipes-extended/shadow/shadow_4.14.0.bb:do_recipe_qa ... (a few more missing do_package objects) =========== 2. When to run the test, and against which commit in poky? RP suggested that the test could run in an antiphase with the nightly a-quick (i.e. 12 hours after). We can do that for a start, but I'm (perhaps prematurely) worrying that this will be unstable: either because the objects from the nightly haven't yet propagated to CDN, or because master has meanwhile (e.g. in the 12 hours that have passed) gained new commits without corresponding cache objects. Are those real concerns? Thoughts? Alex On Fri, 22 Sept 2023 at 19:32, Michael Halstead <mhalst...@linuxfoundation.org> wrote: > > Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN > mirroring proxy endpoint. It's fast and completely free to the project. > However, it will only work for files smaller than 64MB and will send a > redirect to the main server at sstate.yoctoproject.org for files larger than > 64MB. > > The good news is that we have a fast free CDN provided by > https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files. > > Here is an estimated distribution of sstate files currently available: > 1 KB-4 KB: 103168 > 4 KB-16 KB: 1218048 > 16 KB-64 KB: 1886976 > 64 KB-256 KB: 1225216 > 256 KB-1 MB: 170752 > 1 MB-4 MB: 92928 > 4 MB-16 MB: 83200 > 16 MB-64 MB: 40448 > 64 MB-256 MB: 17664 > 256 MB-1 GB: 7936 > 1 GB-4 GB: 2560 > > I'm still working on a CDN solution for the larger files but we can begin > testing https://cdn.jsdelivr.net/yocto/sstate today. > > Here is an example link to a file that is currently popular from each > location: > > http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst > > https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst > > I'm also attaching a list of the largest files in sstate. Anything we can do > to reduce the size of these or to split them into smaller pieces will help > with hosting and with the user's download experience. > > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61381): https://lists.yoctoproject.org/g/yocto/message/61381 Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
