After some workshopping this morning we have a solution for larger files. Almost every package is now available with CDN acceleration. For files over 64MB the fetcher will need to follow a series of redirects to work properly. Files under 64MB will be served fast from every region. Larger files will be proxyed and cached on a smaller network of servers that can handle them. Once cached they will be stored for up to a month. Because sstate is immutable that means fast access after the first use in a region.
Set SSTATE_MIRRORS <https://docs.yoctoproject.org/ref-manual/variables.html#term-SSTATE_MIRRORS> to include https://cdn.jsdelivr.net/yocto/sstate/all or http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report back. If we want the system to perform as fast as possible we need to change SSTATE to work with 64MB file segments but that's a discussion to have after this solution is tested. On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead < mhalst...@linuxfoundation.org> wrote: > Here are packages and size ranges that are not yet served via CDN. > Hopefully testing on the smaller files is possible while we figure out > a solution for these. > > assimp 75M 76M > binutils 64M 100M > boost 68M 148M > ceres-solver 167M 182M > cmake 157M 553M > ffmpeg 68M 69M > gcc 127M 681M > gcc-cross-aarch64 64M 102M > gcc-cross-arm 64M 72M > gcc-cross-i686 65M 80M > gcc-cross-mips 65M 69M > gcc-cross-mips64 64M 69M > gcc-cross-powerpc 64M 72M > gcc-cross-x86_64 65M 104M > gcc-crosssdk-aarch64-pokysdk-linux 65M 75M > gcc-crosssdk-i686-oesdk-linux 65M 77M > gcc-crosssdk-i686-pokysdk-linux 65M 79M > gcc-crosssdk-i686-w64-mingw32 66M 78M > gcc-crosssdk-x86_64-oesdk-linux 65M 78M > gcc-crosssdk-x86_64-pokysdk-linux 65M 80M > gcc-crosssdk-x86_64-w64-mingw32 67M 79M > gdb 66M 105M > ghostscript 64M 115M > git 68M 90M > glibc-locale 81M 228M > go 67M 67M > go-binary-native 82M 82M > go-cross-core2-32 98M 100M > go-cross-core2-64 93M 100M > go-cross-corei7-64 98M 99M > go-cross-cortexa15t2hf-neon 99M 99M > go-cross-cortexa57 93M 99M > go-cross-cortexa8hf-neon 99M 100M > go-cross-mips32r2 98M 99M > go-cross-mips64 99M 99M > go-cross-mips64r2 99M 99M > go-native 86M 86M > go-runtime 64M 82M > grpc 74M 102M > hdf5 78M 137M > influxdb 76M 98M > intel-media-driver 68M 132M > jemalloc 164M 164M > kea 65M 153M > lib32-binutils 64M 70M > lib32-boost 77M 122M > lib32-cmake 158M 305M > lib32-gcc 144M 390M > lib32-gcc-cross-i686 65M 79M > lib32-gcc-cross-mips 65M 69M > lib32-ghostscript 65M 83M > lib32-git 65M 65M > lib32-glibc-locale 81M 228M > lib32-go-cross-x86 98M 99M > lib32-kea 64M 88M > lib32-linux-firmware 275M 419M > lib32-llvm 1126.4M 2662.4M > lib32-ltp 77M 305M > lib32-mesa 65M 68M > lib32-openssl 119M 360M > lib32-piglit 106M 106M > lib32-qemu 275M 639M > lib32-rust 93M 129M > lib32-rust-llvm 64M 76M > lib32-spirv-tools 68M 107M > lib32-valgrind 68M 89M > lib32-vulkan-demos 74M 74M > lib32-vulkan-validation-layers 65M 101M > lib32-webkitgtk 1126.4M 927M > lib64-boost 75M 119M > lib64-gcc 151M 401M > lib64-gcc-cross-mips64 65M 69M > lib64-gcc-cross-x86_64 65M 80M > lib64-glibc-locale 82M 228M > lib64-go-cross-x86_64 98M 99M > lib64-go-runtime 68M 81M > lib64-ltp 67M 347M > lib64-mesa 66M 68M > lib64-openssl 276M 411M > lib64-rust 97M 123M > lib64-rust-llvm 64M 67M > lib64-valgrind 65M 84M > libyang 93M 93M > linux-firmware 275M 419M > linux-intel 120M 395M > linux-yocto 66M 537M > linux-yocto-rt 174M 287M > llvm 1126.4M 3481.6M > llvm-native 71M 88M > ltp 64M 598M > mariadb 1331.2M 966M > mariadb-native 237M 259M > mesa 64M 113M > minifi-cpp 155M 214M > mozjs-102 183M 255M > nativesdk-glibc-locale 81M 226M > nativesdk-qemu 245M 644M > nativesdk-rust 82M 125M > nativesdk-rust-llvm 65M 73M > nodejs 1024.0M 723M > opencv 185M 237M > opengl-es-cts 406M 560M > openssl 92M 685M > php 67M 68M > piglit 106M 106M > poco 75M 107M > python3-grpcio 65M 88M > python3-wxgtk4 70M 100M > qemu 1228.8M 929M > qemu-system-native 65M 71M > renderdoc 67M 82M > rocksdb 177M 272M > rust 85M 140M > rust-llvm 64M 79M > rust-llvm-native 287M 418M > rust-native 78M 85M > spirv-tools 64M 195M > systemd 69M 71M > tesseract-lang 369M 451M > upm 82M 82M > valgrind 64M 128M > vsomeip 82M 149M > vulkan-cts 592M 833M > vulkan-demos 73M 138M > vulkan-validation-layers 65M 108M > webkitgtk 1024.0M 995M > webkitgtk3 1126.4M 1638.4M > wireshark 65M 85M > wxwidgets 70M 88M > > On Fri, Sep 22, 2023 at 10:32 AM Michael Halstead < > mhalst...@linuxfoundation.org> wrote: > >> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN >> mirroring proxy endpoint. It's fast and completely free to the project. >> However, it will only work for files smaller than 64MB and will send a >> redirect to the main server at sstate.yoctoproject.org for files larger >> than 64MB. >> >> The good news is that we have a fast free CDN provided by >> https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files. >> >> Here is an estimated distribution of sstate files currently available: >> 1 KB-4 KB: 103168 >> 4 KB-16 KB: 1218048 >> 16 KB-64 KB: 1886976 >> 64 KB-256 KB: 1225216 >> 256 KB-1 MB: 170752 >> 1 MB-4 MB: 92928 >> 4 MB-16 MB: 83200 >> 16 MB-64 MB: 40448 >> 64 MB-256 MB: 17664 >> 256 MB-1 GB: 7936 >> 1 GB-4 GB: 2560 >> >> I'm still working on a CDN solution for the larger files but we can begin >> testing https://cdn.jsdelivr.net/yocto/sstate today. >> >> Here is an example link to a file that is currently popular from each >> location: >> >> >> http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst >> >> >> https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst >> >> I'm also attaching a list of the largest files in sstate. Anything we can >> do to reduce the size of these or to split them into smaller pieces will >> help with hosting and with the user's download experience. >> >> -- >> Michael Halstead >> Linux Foundation / Yocto Project >> Systems Operations Engineer >> > > > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer > -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61087): https://lists.yoctoproject.org/g/yocto/message/61087 Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
