On Wed, 2023-10-18 at 14:02 +0200, Alexander Kanavin wrote: > On Wed, 18 Oct 2023 at 08:45, Richard Purdie > <richard.pur...@linuxfoundation.org> wrote: > > I'm torn on the targets to test as sato-sdk is a large image and world > > is a lot of work. I'd be tempted to test sato, weston and full-cmdline? > > World is a good test I guess and if from sstate, shouldn't have that > > much of an issue. It does also prove things are working. > > I ran '-S printdiff world' on a blank build directory. First, > scalability isn't great: > > Initialising tasks: 100% > > ##########################################################################################################################################################################| > Time: 0:24:19 > Checking sstate mirror object availability: 100% > > ##################################################################################################################################################| > Time: 0:12:14 > > So it's taking 36 minutes just preparing to fetch the objects, and 2/3 > of that time goes into communicating with hash equivalence server > (e.g. BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687").
FWIW the second time an artefact is pulled from the CDN, it will be much faster. I don't know how much that is a factor here. We know hashequiv is slow, particularly from europe. We've ideas on that we're looking into but for now it is what it is. Part of this work was to find out what the issues are so we write that one up. > Second, there are significant misses. I don't have a clear theory > where they come from, just want to list them: > > The differences between the current build and any cached tasks start > at the following tasks: > /srv/work/alex/poky/meta/recipes-core/meta/meta-ide-support.bb:do_populate_ide_support > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-synaptics_1.9.2.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-devtools/bootchart2/bootchart2_0.14.9.bb:do_create_runtime_spdx > /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-devtools/python/python3-dbusmock_0.29.1.bb:do_create_runtime_spdx > /srv/work/alex/poky/meta/recipes-devtools/dpkg/dpkg_1.22.0.bb:do_configure > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-vmmouse_13.2.0.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-graphics/glew/glew_2.2.0.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-evdev_2.10.6.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-graphics/libva/libva_2.19.0.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-sato/webkit/libwpe_1.14.1.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-meta-base.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-python_1.22.6.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-extended/cups/cups_2.4.6.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-gnome/libdazzle/libdazzle_3.44.0.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_package > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-devtools_1.22.6.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:do_package > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-mouse_1.9.5.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-graphics/waffle/waffle_1.7.2.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-tools_1.2.5.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-rtsp-server_1.22.6.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-devtools/apt/apt_2.6.1.bb:do_install > /srv/work/alex/poky/meta/recipes-gnome/gtk+/gtk4_4.12.3.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-devtools/devel-config/distcc-config.bb:do_create_runtime_spdx > /srv/work/alex/poky/meta/recipes-gnome/libhandy/libhandy_1.8.2.bb:do_collect_spdx_deps > /srv/work/alex/poky/meta/recipes-support/vim/vim_9.0.bb:do_collect_spdx_deps We'll have to dig into this. > So I think we should start with specific images first - minimal, > full-cmdline, weston, sato and sato-sdk are all much faster to check. Agreed, lets start with simpler images, full-cmdline, minimal, sato and weston. We can increase coverage as we move forward (world, sato-sdk). > On qemux86_64 none of them show misses, but on qemuarm64 there are > problems with sato, sato-sdk and weston, i.e. sato-sdk shows: > > The differences between the current build and any cached tasks start > at the following tasks: > /srv/work/alex/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.42.10.bb:do_package_write_rpm > /srv/work/alex/poky/meta/recipes-connectivity/connman/connman-gnome_0.7.bb:do_package_write_rpm > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-examples_1.18.6.bb:do_package_write_rpm > /srv/work/alex/poky/meta/recipes-sato/images/core-image-sato-sdk.bb:do_deploy_source_date_epoch > /srv/work/alex/poky/meta/recipes-graphics/xorg-font/font-util_1.4.1.bb:do_package_write_rpm > /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:do_package > /srv/work/alex/poky/meta/recipes-gnome/librsvg/librsvg_2.56.3.bb:do_package_write_rpm > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.22.6.bb:do_package_write_rpm > > > I'm not sure if that's because cache for the current master needs to > be populated properly first, or if there's a deeper issue. We made a release build with current master so the cache should be good. We're going to have to look into it. > > Either we start logging what we've built so we get the last known > > revisions, or we run the test as part of a-quick/a-full at the end of > > the build? I don't really want to extend the build but I'm not sure we > > may have much choice. > > I think it can be a selftest and bundled into > trigger-build-posttrigger which already runs > 'buildoptions.SourceMirroring.test_yocto_source_mirror': > > https://git.yoctoproject.org/yocto-autobuilder-helper/tree/config.json#n465 > > The various yocto mirrors tests could be tagged in a way similar to > machine/toolchain selftests, so that they're excluded from regular > oe-selftest, but run specifically here. That sounds reasonable. oe-selftest has the concept of "tags" so we should create some for these tests and use those as the hardcoded lists are a pain. We already do that for the toolchain-user, toolchain-system and machine ones iirc. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61403): https://lists.yoctoproject.org/g/yocto/message/61403 Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
