[yocto] CDN for sstate.yoctoproject.org

[Michael Halstead]

Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN
mirroring proxy endpoint. It's fast and completely free to the project.
However, it will only work for files smaller than 64MB and will send a
redirect to the main server at sstate.yoctoproject.org for files larger
than 64MB.

The good news is that we have a fast free CDN provided by
https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files.

Here is an estimated distribution of sstate files currently available:
1 KB-4 KB: 103168
4 KB-16 KB: 1218048
16 KB-64 KB: 1886976
64 KB-256 KB: 1225216
256 KB-1 MB: 170752
1 MB-4 MB: 92928
4 MB-16 MB: 83200
16 MB-64 MB: 40448
64 MB-256 MB: 17664
256 MB-1 GB: 7936
1 GB-4 GB: 2560

I'm still working on a CDN solution for the larger files but we can begin
testing https://cdn.jsdelivr.net/yocto/sstate today.

Here is an example link to a file that is currently popular from each
location:

http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst

https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst

I'm also attaching a list of the largest files in sstate. Anything we can
do to reduce the size of these or to split them into smaller pieces will
help with hosting and with the user's download experience.

-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer
1.5G
./a8/b4/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:a8b46f6a3a1bc1cf2b87868b874cb5fdee215e54ec9c0137e158928338a67929_package.tar.zst
1.5G
./c4/dd/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:c4dd7c7b4d30a76d8a6dc949b358a9c36a6ee0b51f767c2d212c9de17fc3f642_package.tar.zst
1.5G
./a3/65/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:a365274c552d9522255d5ecb00772c66a0c247c564f640a6f306e6223996dbe1_package.tar.zst
1.5G
./8b/55/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:8b55b88056e5949753a2e8c22a05c6ed0e67ba7a62057b0af1c85039af708e57_package.tar.zst
1.5G
./3e/fd/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:3efd12902f0bd914a06baa52ba8ea3dc5245b50a72d1f5be230a785953430050_package.tar.zst
1.5G
./51/a8/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:51a8aac859e1fe22e41a0cbf3bc7f4b6be5f2d6083890a0c1ad30ef04cbd562a_package.tar.zst
1.5G
./be/cc/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:becc86252d869953af9a0551a714eecb8293b2003f055240f3d62a08cf6fb587_package.tar.zst
1.5G
./11/38/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:11382ddafe9d877b423368098b3767cdb76997596258ac0b5abe84e039de792a_package.tar.zst
1.5G
./65/f3/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:65f364cf45aa1cdfad4ab2a06ac83f8c5192e50e934a5c80f3b3ece9e13e9d7f_package.tar.zst
1.5G
./d8/59/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:d859b97500df7b9583005d5d133f756ad9993ced0d87b3a3374ab8a284ed5203_package.tar.zst
1.5G
./5c/17/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:5c1791c3fb9aa770d6fe841c1edcaf92acf5036df04fef64d0d1f920bc12123e_package.tar.zst
1.5G
./35/45/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:3545516b619f151bcf2f284ee0aafe6026703f260c2637a5903c3045b61d786a_package.tar.zst
1.5G
./0f/60/sstate:webkitgtk:core2-64-poky-linux-musl:2.40.5:r0:core2-64:11:0f60d7aa20cb7ff17dc3fc2253f084e1c2619ce9a271ebe63d0a33d0eefacc9d_package.tar.zst
1.5G
./78/8c/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:788c4637e942d30a42bc5cc091ee8bf09bed90dc88ad97906e6cbdf4f69ba478_package.tar.zst
1.5G
./f2/98/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:f2982a32c5eb50c35ab6ff911f3e6a425dcaba54dc05a8d7acb33e49ce001762_package.tar.zst
1.5G
./ed/91/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:ed91adec9fab6be660905c14573d83b29edc638142a696ea10a84f159b97feab_package.tar.zst
1.5G
./f3/7e/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:f37eb7f6f9a0b5a5e892b9aaedb0cd06dc3448427b4a60390c47a424d5569bd8_package.tar.zst
1.5G
./5c/a6/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:5ca63e9f225aa53ce2e4d23aea4d80610f381aec9fc66d34b6117b0421dad349_package.tar.zst
1.5G
./bf/db/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:bfdbea005a7941b35b5dc182763c59b19e84f3fb06a60692351358416ebf4106_package.tar.zst
1.5G
./30/88/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:3088790a9fb368501eab1135fce08fc4653cc0c4c3e7979cf3a0bc5730549ae2_package.tar.zst
1.5G
./6b/c9/sstate:webkitgtk:core2-64-poky-linux:2.40.5:r0:core2-64:11:6bc94a716ef0da710d58e55bbefd7645d59918edbb4dc4b8b367452020540c6e_package.tar.zst
1.5G
./e9/02/ssta

Re: [yocto] CDN for sstate.yoctoproject.org

[Michael Halstead]

Here are packages and size ranges that are not yet served via CDN.
Hopefully testing on the smaller files is possible while we figure out
a solution for these.

assimp 75M 76M
binutils 64M 100M
boost 68M 148M
ceres-solver 167M 182M
cmake 157M 553M
ffmpeg 68M 69M
gcc 127M 681M
gcc-cross-aarch64 64M 102M
gcc-cross-arm 64M 72M
gcc-cross-i686 65M 80M
gcc-cross-mips 65M 69M
gcc-cross-mips64 64M 69M
gcc-cross-powerpc 64M 72M
gcc-cross-x86_64 65M 104M
gcc-crosssdk-aarch64-pokysdk-linux 65M 75M
gcc-crosssdk-i686-oesdk-linux 65M 77M
gcc-crosssdk-i686-pokysdk-linux 65M 79M
gcc-crosssdk-i686-w64-mingw32 66M 78M
gcc-crosssdk-x86_64-oesdk-linux 65M 78M
gcc-crosssdk-x86_64-pokysdk-linux 65M 80M
gcc-crosssdk-x86_64-w64-mingw32 67M 79M
gdb 66M 105M
ghostscript 64M 115M
git 68M 90M
glibc-locale 81M 228M
go 67M 67M
go-binary-native 82M 82M
go-cross-core2-32 98M 100M
go-cross-core2-64 93M 100M
go-cross-corei7-64 98M 99M
go-cross-cortexa15t2hf-neon 99M 99M
go-cross-cortexa57 93M 99M
go-cross-cortexa8hf-neon 99M 100M
go-cross-mips32r2 98M 99M
go-cross-mips64 99M 99M
go-cross-mips64r2 99M 99M
go-native 86M 86M
go-runtime 64M 82M
grpc 74M 102M
hdf5 78M 137M
influxdb 76M 98M
intel-media-driver 68M 132M
jemalloc 164M 164M
kea 65M 153M
lib32-binutils 64M 70M
lib32-boost 77M 122M
lib32-cmake 158M 305M
lib32-gcc 144M 390M
lib32-gcc-cross-i686 65M 79M
lib32-gcc-cross-mips 65M 69M
lib32-ghostscript 65M 83M
lib32-git 65M 65M
lib32-glibc-locale 81M 228M
lib32-go-cross-x86 98M 99M
lib32-kea 64M 88M
lib32-linux-firmware 275M 419M
lib32-llvm 1126.4M 2662.4M
lib32-ltp 77M 305M
lib32-mesa 65M 68M
lib32-openssl 119M 360M
lib32-piglit 106M 106M
lib32-qemu 275M 639M
lib32-rust 93M 129M
lib32-rust-llvm 64M 76M
lib32-spirv-tools 68M 107M
lib32-valgrind 68M 89M
lib32-vulkan-demos 74M 74M
lib32-vulkan-validation-layers 65M 101M
lib32-webkitgtk 1126.4M 927M
lib64-boost 75M 119M
lib64-gcc 151M 401M
lib64-gcc-cross-mips64 65M 69M
lib64-gcc-cross-x86_64 65M 80M
lib64-glibc-locale 82M 228M
lib64-go-cross-x86_64 98M 99M
lib64-go-runtime 68M 81M
lib64-ltp 67M 347M
lib64-mesa 66M 68M
lib64-openssl 276M 411M
lib64-rust 97M 123M
lib64-rust-llvm 64M 67M
lib64-valgrind 65M 84M
libyang 93M 93M
linux-firmware 275M 419M
linux-intel 120M 395M
linux-yocto 66M 537M
linux-yocto-rt 174M 287M
llvm 1126.4M 3481.6M
llvm-native 71M 88M
ltp 64M 598M
mariadb 1331.2M 966M
mariadb-native 237M 259M
mesa 64M 113M
minifi-cpp 155M 214M
mozjs-102 183M 255M
nativesdk-glibc-locale 81M 226M
nativesdk-qemu 245M 644M
nativesdk-rust 82M 125M
nativesdk-rust-llvm 65M 73M
nodejs 1024.0M 723M
opencv 185M 237M
opengl-es-cts 406M 560M
openssl 92M 685M
php 67M 68M
piglit 106M 106M
poco 75M 107M
python3-grpcio 65M 88M
python3-wxgtk4 70M 100M
qemu 1228.8M 929M
qemu-system-native 65M 71M
renderdoc 67M 82M
rocksdb 177M 272M
rust 85M 140M
rust-llvm 64M 79M
rust-llvm-native 287M 418M
rust-native 78M 85M
spirv-tools 64M 195M
systemd 69M 71M
tesseract-lang 369M 451M
upm 82M 82M
valgrind 64M 128M
vsomeip 82M 149M
vulkan-cts 592M 833M
vulkan-demos 73M 138M
vulkan-validation-layers 65M 108M
webkitgtk 1024.0M 995M
webkitgtk3 1126.4M 1638.4M
wireshark 65M 85M
wxwidgets 70M 88M

On Fri, Sep 22, 2023 at 10:32 AM Michael Halstead <
mhalst...@linuxfoundation.org> wrote:

> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN
> mirroring proxy endpoint. It's fast and completely free to the project.
> However, it will only work for files smaller than 64MB and will send a
> redirect to the main server at sstate.yoctoproject.org for files larger
> than 64MB.
>
> The good news is that we have a fast free CDN provided by
> https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files.
>
> Here is an estimated distribution of sstate files currently available:
> 1 KB-4 KB: 103168
> 4 KB-16 KB: 1218048
> 16 KB-64 KB: 1886976
> 64 KB-256 KB: 1225216
> 256 KB-1 MB: 170752
> 1 MB-4 MB: 92928
> 4 MB-16 MB: 83200
> 16 MB-64 MB: 40448
> 64 MB-256 MB: 17664
> 256 MB-1 GB: 7936
> 1 GB-4 GB: 2560
>
> I'm still working on a CDN solution for the larger files but we can begin
> testing https://cdn.jsdelivr.net/yocto/sstate today.
>
> Here is an example link to a file that is currently popular from each
> location:
>
>
> http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst
>
>
> https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst
>
> I'm also attaching a list of the largest files in sstate. Anything we can
> do to reduce the size of these or to split them into smaller pieces will
> help with hosting and with the user's download experience.
>
> --
> Michael Halstead
> Linux Foundation / Yocto Project
> Systems Operations Engineer
>


-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engine

Re: [yocto] CDN for sstate.yoctoproject.org

[Michael Halstead]

After some workshopping this morning we have a solution for larger files.
Almost every package is now available with CDN acceleration. For files over
64MB the fetcher will need to follow a series of redirects to work
properly. Files under 64MB will be served fast from every region. Larger
files will be proxyed and cached on a smaller network of servers that can
handle them. Once cached they will be stored for up to a month. Because
sstate is immutable that means fast access after the first use in a region.

Set SSTATE_MIRRORS

to include https://cdn.jsdelivr.net/yocto/sstate/all or
http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report back.

If we want the system to perform as fast as possible we need to change
SSTATE to work with 64MB file segments but that's a discussion to have
after this solution is tested.


On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead <
mhalst...@linuxfoundation.org> wrote:

> Here are packages and size ranges that are not yet served via CDN.
> Hopefully testing on the smaller files is possible while we figure out
> a solution for these.
>
> assimp 75M 76M
> binutils 64M 100M
> boost 68M 148M
> ceres-solver 167M 182M
> cmake 157M 553M
> ffmpeg 68M 69M
> gcc 127M 681M
> gcc-cross-aarch64 64M 102M
> gcc-cross-arm 64M 72M
> gcc-cross-i686 65M 80M
> gcc-cross-mips 65M 69M
> gcc-cross-mips64 64M 69M
> gcc-cross-powerpc 64M 72M
> gcc-cross-x86_64 65M 104M
> gcc-crosssdk-aarch64-pokysdk-linux 65M 75M
> gcc-crosssdk-i686-oesdk-linux 65M 77M
> gcc-crosssdk-i686-pokysdk-linux 65M 79M
> gcc-crosssdk-i686-w64-mingw32 66M 78M
> gcc-crosssdk-x86_64-oesdk-linux 65M 78M
> gcc-crosssdk-x86_64-pokysdk-linux 65M 80M
> gcc-crosssdk-x86_64-w64-mingw32 67M 79M
> gdb 66M 105M
> ghostscript 64M 115M
> git 68M 90M
> glibc-locale 81M 228M
> go 67M 67M
> go-binary-native 82M 82M
> go-cross-core2-32 98M 100M
> go-cross-core2-64 93M 100M
> go-cross-corei7-64 98M 99M
> go-cross-cortexa15t2hf-neon 99M 99M
> go-cross-cortexa57 93M 99M
> go-cross-cortexa8hf-neon 99M 100M
> go-cross-mips32r2 98M 99M
> go-cross-mips64 99M 99M
> go-cross-mips64r2 99M 99M
> go-native 86M 86M
> go-runtime 64M 82M
> grpc 74M 102M
> hdf5 78M 137M
> influxdb 76M 98M
> intel-media-driver 68M 132M
> jemalloc 164M 164M
> kea 65M 153M
> lib32-binutils 64M 70M
> lib32-boost 77M 122M
> lib32-cmake 158M 305M
> lib32-gcc 144M 390M
> lib32-gcc-cross-i686 65M 79M
> lib32-gcc-cross-mips 65M 69M
> lib32-ghostscript 65M 83M
> lib32-git 65M 65M
> lib32-glibc-locale 81M 228M
> lib32-go-cross-x86 98M 99M
> lib32-kea 64M 88M
> lib32-linux-firmware 275M 419M
> lib32-llvm 1126.4M 2662.4M
> lib32-ltp 77M 305M
> lib32-mesa 65M 68M
> lib32-openssl 119M 360M
> lib32-piglit 106M 106M
> lib32-qemu 275M 639M
> lib32-rust 93M 129M
> lib32-rust-llvm 64M 76M
> lib32-spirv-tools 68M 107M
> lib32-valgrind 68M 89M
> lib32-vulkan-demos 74M 74M
> lib32-vulkan-validation-layers 65M 101M
> lib32-webkitgtk 1126.4M 927M
> lib64-boost 75M 119M
> lib64-gcc 151M 401M
> lib64-gcc-cross-mips64 65M 69M
> lib64-gcc-cross-x86_64 65M 80M
> lib64-glibc-locale 82M 228M
> lib64-go-cross-x86_64 98M 99M
> lib64-go-runtime 68M 81M
> lib64-ltp 67M 347M
> lib64-mesa 66M 68M
> lib64-openssl 276M 411M
> lib64-rust 97M 123M
> lib64-rust-llvm 64M 67M
> lib64-valgrind 65M 84M
> libyang 93M 93M
> linux-firmware 275M 419M
> linux-intel 120M 395M
> linux-yocto 66M 537M
> linux-yocto-rt 174M 287M
> llvm 1126.4M 3481.6M
> llvm-native 71M 88M
> ltp 64M 598M
> mariadb 1331.2M 966M
> mariadb-native 237M 259M
> mesa 64M 113M
> minifi-cpp 155M 214M
> mozjs-102 183M 255M
> nativesdk-glibc-locale 81M 226M
> nativesdk-qemu 245M 644M
> nativesdk-rust 82M 125M
> nativesdk-rust-llvm 65M 73M
> nodejs 1024.0M 723M
> opencv 185M 237M
> opengl-es-cts 406M 560M
> openssl 92M 685M
> php 67M 68M
> piglit 106M 106M
> poco 75M 107M
> python3-grpcio 65M 88M
> python3-wxgtk4 70M 100M
> qemu 1228.8M 929M
> qemu-system-native 65M 71M
> renderdoc 67M 82M
> rocksdb 177M 272M
> rust 85M 140M
> rust-llvm 64M 79M
> rust-llvm-native 287M 418M
> rust-native 78M 85M
> spirv-tools 64M 195M
> systemd 69M 71M
> tesseract-lang 369M 451M
> upm 82M 82M
> valgrind 64M 128M
> vsomeip 82M 149M
> vulkan-cts 592M 833M
> vulkan-demos 73M 138M
> vulkan-validation-layers 65M 108M
> webkitgtk 1024.0M 995M
> webkitgtk3 1126.4M 1638.4M
> wireshark 65M 85M
> wxwidgets 70M 88M
>
> On Fri, Sep 22, 2023 at 10:32 AM Michael Halstead <
> mhalst...@linuxfoundation.org> wrote:
>
>> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN
>> mirroring proxy endpoint. It's fast and completely free to the project.
>> However, it will only work for files smaller than 64MB and will send a
>> redirect to the main server at sstate.yoctoproject.org for files larger
>> than 64MB.
>>
>> The good news is that we have a fast free CDN provided by
>> https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files.
>>
>> Here is an estimated distribution of s

Re: [yocto] CDN for sstate.yoctoproject.org

[Michael Halstead]

When adding https://cdn.jsdelivr.net/yocto/sstate/all please remove any
reference to sstate.yoctoproject.org from SSTATE_MIRRORS to ensure a proper
test.

Thank you

On Sat, Sep 23, 2023 at 12:51 PM Michael Halstead <
mhalst...@linuxfoundation.org> wrote:

> After some workshopping this morning we have a solution for larger files.
> Almost every package is now available with CDN acceleration. For files over
> 64MB the fetcher will need to follow a series of redirects to work
> properly. Files under 64MB will be served fast from every region. Larger
> files will be proxyed and cached on a smaller network of servers that can
> handle them. Once cached they will be stored for up to a month. Because
> sstate is immutable that means fast access after the first use in a region.
>
> Set SSTATE_MIRRORS
> 
> to include https://cdn.jsdelivr.net/yocto/sstate/all or
> http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report back.
>
> If we want the system to perform as fast as possible we need to change
> SSTATE to work with 64MB file segments but that's a discussion to have
> after this solution is tested.
>
>
> On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead <
> mhalst...@linuxfoundation.org> wrote:
>
>> Here are packages and size ranges that are not yet served via CDN.
>> Hopefully testing on the smaller files is possible while we figure out
>> a solution for these.
>>
>> assimp 75M 76M
>> binutils 64M 100M
>> boost 68M 148M
>> ceres-solver 167M 182M
>> cmake 157M 553M
>> ffmpeg 68M 69M
>> gcc 127M 681M
>> gcc-cross-aarch64 64M 102M
>> gcc-cross-arm 64M 72M
>> gcc-cross-i686 65M 80M
>> gcc-cross-mips 65M 69M
>> gcc-cross-mips64 64M 69M
>> gcc-cross-powerpc 64M 72M
>> gcc-cross-x86_64 65M 104M
>> gcc-crosssdk-aarch64-pokysdk-linux 65M 75M
>> gcc-crosssdk-i686-oesdk-linux 65M 77M
>> gcc-crosssdk-i686-pokysdk-linux 65M 79M
>> gcc-crosssdk-i686-w64-mingw32 66M 78M
>> gcc-crosssdk-x86_64-oesdk-linux 65M 78M
>> gcc-crosssdk-x86_64-pokysdk-linux 65M 80M
>> gcc-crosssdk-x86_64-w64-mingw32 67M 79M
>> gdb 66M 105M
>> ghostscript 64M 115M
>> git 68M 90M
>> glibc-locale 81M 228M
>> go 67M 67M
>> go-binary-native 82M 82M
>> go-cross-core2-32 98M 100M
>> go-cross-core2-64 93M 100M
>> go-cross-corei7-64 98M 99M
>> go-cross-cortexa15t2hf-neon 99M 99M
>> go-cross-cortexa57 93M 99M
>> go-cross-cortexa8hf-neon 99M 100M
>> go-cross-mips32r2 98M 99M
>> go-cross-mips64 99M 99M
>> go-cross-mips64r2 99M 99M
>> go-native 86M 86M
>> go-runtime 64M 82M
>> grpc 74M 102M
>> hdf5 78M 137M
>> influxdb 76M 98M
>> intel-media-driver 68M 132M
>> jemalloc 164M 164M
>> kea 65M 153M
>> lib32-binutils 64M 70M
>> lib32-boost 77M 122M
>> lib32-cmake 158M 305M
>> lib32-gcc 144M 390M
>> lib32-gcc-cross-i686 65M 79M
>> lib32-gcc-cross-mips 65M 69M
>> lib32-ghostscript 65M 83M
>> lib32-git 65M 65M
>> lib32-glibc-locale 81M 228M
>> lib32-go-cross-x86 98M 99M
>> lib32-kea 64M 88M
>> lib32-linux-firmware 275M 419M
>> lib32-llvm 1126.4M 2662.4M
>> lib32-ltp 77M 305M
>> lib32-mesa 65M 68M
>> lib32-openssl 119M 360M
>> lib32-piglit 106M 106M
>> lib32-qemu 275M 639M
>> lib32-rust 93M 129M
>> lib32-rust-llvm 64M 76M
>> lib32-spirv-tools 68M 107M
>> lib32-valgrind 68M 89M
>> lib32-vulkan-demos 74M 74M
>> lib32-vulkan-validation-layers 65M 101M
>> lib32-webkitgtk 1126.4M 927M
>> lib64-boost 75M 119M
>> lib64-gcc 151M 401M
>> lib64-gcc-cross-mips64 65M 69M
>> lib64-gcc-cross-x86_64 65M 80M
>> lib64-glibc-locale 82M 228M
>> lib64-go-cross-x86_64 98M 99M
>> lib64-go-runtime 68M 81M
>> lib64-ltp 67M 347M
>> lib64-mesa 66M 68M
>> lib64-openssl 276M 411M
>> lib64-rust 97M 123M
>> lib64-rust-llvm 64M 67M
>> lib64-valgrind 65M 84M
>> libyang 93M 93M
>> linux-firmware 275M 419M
>> linux-intel 120M 395M
>> linux-yocto 66M 537M
>> linux-yocto-rt 174M 287M
>> llvm 1126.4M 3481.6M
>> llvm-native 71M 88M
>> ltp 64M 598M
>> mariadb 1331.2M 966M
>> mariadb-native 237M 259M
>> mesa 64M 113M
>> minifi-cpp 155M 214M
>> mozjs-102 183M 255M
>> nativesdk-glibc-locale 81M 226M
>> nativesdk-qemu 245M 644M
>> nativesdk-rust 82M 125M
>> nativesdk-rust-llvm 65M 73M
>> nodejs 1024.0M 723M
>> opencv 185M 237M
>> opengl-es-cts 406M 560M
>> openssl 92M 685M
>> php 67M 68M
>> piglit 106M 106M
>> poco 75M 107M
>> python3-grpcio 65M 88M
>> python3-wxgtk4 70M 100M
>> qemu 1228.8M 929M
>> qemu-system-native 65M 71M
>> renderdoc 67M 82M
>> rocksdb 177M 272M
>> rust 85M 140M
>> rust-llvm 64M 79M
>> rust-llvm-native 287M 418M
>> rust-native 78M 85M
>> spirv-tools 64M 195M
>> systemd 69M 71M
>> tesseract-lang 369M 451M
>> upm 82M 82M
>> valgrind 64M 128M
>> vsomeip 82M 149M
>> vulkan-cts 592M 833M
>> vulkan-demos 73M 138M
>> vulkan-validation-layers 65M 108M
>> webkitgtk 1024.0M 995M
>> webkitgtk3 1126.4M 1638.4M
>> wireshark 65M 85M
>> wxwidgets 70M 88M
>>
>> On Fri, Sep 22, 2023 at 10:32 AM Michael Halstead <
>> mhalst...@linuxfoundation.org> wrote:
>>
>>> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up 

Re: [yocto] CDN for sstate.yoctoproject.org

[Khem Raj]

For fun, I tried scratch build of core-image-sato-sdk from master-next
+ my changes with the new SSTATE_MIRROR

Initialising tasks: 100%
|###|
Time: 0:02:16
Checking sstate mirror object availability: 100%
|###|
Time: 0:02:26
Sstate summary: Wanted 5040 Local 3 Mirrors 5024 Missed 13 Current 0
(99% match, 0% complete)

...

NOTE: Tasks Summary: Attempted 10124 tasks of which 10092 didn't need
to be rerun and all succeeded.

So match percentage is cool.

Elapsed time: 799.86 seconds

which is approx 13 mins so pretty cool given that sstate download is
also included in this time.

On Sat, Sep 23, 2023 at 12:53 PM Michael Halstead
 wrote:
>
> When adding https://cdn.jsdelivr.net/yocto/sstate/all please remove any 
> reference to sstate.yoctoproject.org from SSTATE_MIRRORS to ensure a proper 
> test.
>
> Thank you
>
> On Sat, Sep 23, 2023 at 12:51 PM Michael Halstead 
>  wrote:
>>
>> After some workshopping this morning we have a solution for larger files. 
>> Almost every package is now available with CDN acceleration. For files over 
>> 64MB the fetcher will need to follow a series of redirects to work properly. 
>> Files under 64MB will be served fast from every region. Larger files will be 
>> proxyed and cached on a smaller network of servers that can handle them. 
>> Once cached they will be stored for up to a month. Because sstate is 
>> immutable that means fast access after the first use in a region.
>>
>> Set SSTATE_MIRRORS to include https://cdn.jsdelivr.net/yocto/sstate/all or 
>> http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report back.
>>
>> If we want the system to perform as fast as possible we need to change 
>> SSTATE to work with 64MB file segments but that's a discussion to have after 
>> this solution is tested.
>>
>>
>> On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead 
>>  wrote:
>>>
>>> Here are packages and size ranges that are not yet served via CDN. 
>>> Hopefully testing on the smaller files is possible while we figure out a 
>>> solution for these.
>>>
>>> assimp 75M 76M
>>> binutils 64M 100M
>>> boost 68M 148M
>>> ceres-solver 167M 182M
>>> cmake 157M 553M
>>> ffmpeg 68M 69M
>>> gcc 127M 681M
>>> gcc-cross-aarch64 64M 102M
>>> gcc-cross-arm 64M 72M
>>> gcc-cross-i686 65M 80M
>>> gcc-cross-mips 65M 69M
>>> gcc-cross-mips64 64M 69M
>>> gcc-cross-powerpc 64M 72M
>>> gcc-cross-x86_64 65M 104M
>>> gcc-crosssdk-aarch64-pokysdk-linux 65M 75M
>>> gcc-crosssdk-i686-oesdk-linux 65M 77M
>>> gcc-crosssdk-i686-pokysdk-linux 65M 79M
>>> gcc-crosssdk-i686-w64-mingw32 66M 78M
>>> gcc-crosssdk-x86_64-oesdk-linux 65M 78M
>>> gcc-crosssdk-x86_64-pokysdk-linux 65M 80M
>>> gcc-crosssdk-x86_64-w64-mingw32 67M 79M
>>> gdb 66M 105M
>>> ghostscript 64M 115M
>>> git 68M 90M
>>> glibc-locale 81M 228M
>>> go 67M 67M
>>> go-binary-native 82M 82M
>>> go-cross-core2-32 98M 100M
>>> go-cross-core2-64 93M 100M
>>> go-cross-corei7-64 98M 99M
>>> go-cross-cortexa15t2hf-neon 99M 99M
>>> go-cross-cortexa57 93M 99M
>>> go-cross-cortexa8hf-neon 99M 100M
>>> go-cross-mips32r2 98M 99M
>>> go-cross-mips64 99M 99M
>>> go-cross-mips64r2 99M 99M
>>> go-native 86M 86M
>>> go-runtime 64M 82M
>>> grpc 74M 102M
>>> hdf5 78M 137M
>>> influxdb 76M 98M
>>> intel-media-driver 68M 132M
>>> jemalloc 164M 164M
>>> kea 65M 153M
>>> lib32-binutils 64M 70M
>>> lib32-boost 77M 122M
>>> lib32-cmake 158M 305M
>>> lib32-gcc 144M 390M
>>> lib32-gcc-cross-i686 65M 79M
>>> lib32-gcc-cross-mips 65M 69M
>>> lib32-ghostscript 65M 83M
>>> lib32-git 65M 65M
>>> lib32-glibc-locale 81M 228M
>>> lib32-go-cross-x86 98M 99M
>>> lib32-kea 64M 88M
>>> lib32-linux-firmware 275M 419M
>>> lib32-llvm 1126.4M 2662.4M
>>> lib32-ltp 77M 305M
>>> lib32-mesa 65M 68M
>>> lib32-openssl 119M 360M
>>> lib32-piglit 106M 106M
>>> lib32-qemu 275M 639M
>>> lib32-rust 93M 129M
>>> lib32-rust-llvm 64M 76M
>>> lib32-spirv-tools 68M 107M
>>> lib32-valgrind 68M 89M
>>> lib32-vulkan-demos 74M 74M
>>> lib32-vulkan-validation-layers 65M 101M
>>> lib32-webkitgtk 1126.4M 927M
>>> lib64-boost 75M 119M
>>> lib64-gcc 151M 401M
>>> lib64-gcc-cross-mips64 65M 69M
>>> lib64-gcc-cross-x86_64 65M 80M
>>> lib64-glibc-locale 82M 228M
>>> lib64-go-cross-x86_64 98M 99M
>>> lib64-go-runtime 68M 81M
>>> lib64-ltp 67M 347M
>>> lib64-mesa 66M 68M
>>> lib64-openssl 276M 411M
>>> lib64-rust 97M 123M
>>> lib64-rust-llvm 64M 67M
>>> lib64-valgrind 65M 84M
>>> libyang 93M 93M
>>> linux-firmware 275M 419M
>>> linux-intel 120M 395M
>>> linux-yocto 66M 537M
>>> linux-yocto-rt 174M 287M
>>> llvm 1126.4M 3481.6M
>>> llvm-native 71M 88M
>>> ltp 64M 598M
>>> mariadb 1331.2M 966M
>>> mariadb-native 237M 259M
>>> mesa 64M 113M
>>> minifi-cpp 155M 214M
>>> mozjs-102 183M 255M
>>> nativesdk-glibc-locale 81M 226M
>>> na

Re: [yocto] CDN for sstate.yoctoproject.org

[Armin Kuster]

Hello Micheal,

Thanks for working on this.

BR,
Armin

On 9/23/23 3:52 PM, Michael Halstead wrote:
When adding https://cdn.jsdelivr.net/yocto/sstate/all 
 please remove any 
reference to sstate.yoctoproject.org  
from SSTATE_MIRRORS to ensure a proper test.


Thank you

On Sat, Sep 23, 2023 at 12:51 PM Michael Halstead 
 wrote:


After some workshopping this morning we have a solution for larger
files. Almost every package is now available with CDN
acceleration. For files over 64MB the fetcher will need to follow
a series of redirects to work properly. Files under 64MB will be
served fast from every region. Larger files will be proxyed and
cached on a smaller network of servers that can handle them. Once
cached they will be stored for up to a month. Because sstate is
immutable that means fast access after the first use in a region.

Set SSTATE_MIRRORS


to include https://cdn.jsdelivr.net/yocto/sstate/all or
http://cdn.jsdelivr.net/yocto/sstate/all, test it out, and report
back.

If we want the system to perform as fast as possible we need to
change SSTATE to work with 64MB file segments but that's a
discussion to have after this solution is tested.


On Fri, Sep 22, 2023 at 12:03 PM Michael Halstead
 wrote:

Here are packages and size ranges that are not yet served via
CDN. Hopefully testing on the smaller files is possible while
we figure out a solution for these.

assimp 75M 76M
binutils 64M 100M
boost 68M 148M
ceres-solver 167M 182M
cmake 157M 553M
ffmpeg 68M 69M
gcc 127M 681M
gcc-cross-aarch64 64M 102M
gcc-cross-arm 64M 72M
gcc-cross-i686 65M 80M
gcc-cross-mips 65M 69M
gcc-cross-mips64 64M 69M
gcc-cross-powerpc 64M 72M
gcc-cross-x86_64 65M 104M
gcc-crosssdk-aarch64-pokysdk-linux 65M 75M
gcc-crosssdk-i686-oesdk-linux 65M 77M
gcc-crosssdk-i686-pokysdk-linux 65M 79M
gcc-crosssdk-i686-w64-mingw32 66M 78M
gcc-crosssdk-x86_64-oesdk-linux 65M 78M
gcc-crosssdk-x86_64-pokysdk-linux 65M 80M
gcc-crosssdk-x86_64-w64-mingw32 67M 79M
gdb 66M 105M
ghostscript 64M 115M
git 68M 90M
glibc-locale 81M 228M
go 67M 67M
go-binary-native 82M 82M
go-cross-core2-32 98M 100M
go-cross-core2-64 93M 100M
go-cross-corei7-64 98M 99M
go-cross-cortexa15t2hf-neon 99M 99M
go-cross-cortexa57 93M 99M
go-cross-cortexa8hf-neon 99M 100M
go-cross-mips32r2 98M 99M
go-cross-mips64 99M 99M
go-cross-mips64r2 99M 99M
go-native 86M 86M
go-runtime 64M 82M
grpc 74M 102M
hdf5 78M 137M
influxdb 76M 98M
intel-media-driver 68M 132M
jemalloc 164M 164M
kea 65M 153M
lib32-binutils 64M 70M
lib32-boost 77M 122M
lib32-cmake 158M 305M
lib32-gcc 144M 390M
lib32-gcc-cross-i686 65M 79M
lib32-gcc-cross-mips 65M 69M
lib32-ghostscript 65M 83M
lib32-git 65M 65M
lib32-glibc-locale 81M 228M
lib32-go-cross-x86 98M 99M
lib32-kea 64M 88M
lib32-linux-firmware 275M 419M
lib32-llvm 1126.4M 2662.4M
lib32-ltp 77M 305M
lib32-mesa 65M 68M
lib32-openssl 119M 360M
lib32-piglit 106M 106M
lib32-qemu 275M 639M
lib32-rust 93M 129M
lib32-rust-llvm 64M 76M
lib32-spirv-tools 68M 107M
lib32-valgrind 68M 89M
lib32-vulkan-demos 74M 74M
lib32-vulkan-validation-layers 65M 101M
lib32-webkitgtk 1126.4M 927M
lib64-boost 75M 119M
lib64-gcc 151M 401M
lib64-gcc-cross-mips64 65M 69M
lib64-gcc-cross-x86_64 65M 80M
lib64-glibc-locale 82M 228M
lib64-go-cross-x86_64 98M 99M
lib64-go-runtime 68M 81M
lib64-ltp 67M 347M
lib64-mesa 66M 68M
lib64-openssl 276M 411M
lib64-rust 97M 123M
lib64-rust-llvm 64M 67M
lib64-valgrind 65M 84M
libyang 93M 93M
linux-firmware 275M 419M
linux-intel 120M 395M
linux-yocto 66M 537M
linux-yocto-rt 174M 287M
llvm 1126.4M 3481.6M
llvm-native 71M 88M
ltp 64M 598M
mariadb 1331.2M 966M
mariadb-native 237M 259M
mesa 64M 113M
minifi-cpp 155M 214M
mozjs-102 183M 255M
nativesdk-glibc-locale 81M 226M
nativesdk-qemu 245M 644M
nativesdk-rust 82M 125M
nativesdk-rust-llvm 65M 73M
nodejs 1024.0M 723M
opencv 185M 237M
opengl-es-cts 406M 560M
openssl 92M 685M
php 67M 68M
piglit 106M 106M
poco 75M 107M
python3-gr

Re: [yocto] CDN for sstate.yoctoproject.org

[Alexander Kanavin]

Thanks for working on this! I finally got to playing with CDN mirror a
bit, and it seems to basically work ok, so next I'm going to write AB
tests that check that it remains useful. Specifically:

1. What should the test be?

I tried 'bitbake -S printdiff core-image-sato' on a blank build
directory, and the result against current master is:

===
Checking sstate mirror object availability: 100%
|##|
Time: 0:05:35
The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-sato/images/core-image-sato.bb:do_deploy_source_date_epoch
/srv/work/alex/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_install
===

I think this is pretty good! The two missing objects depend on the
current date, and so should go to the exception list in the test, and
otherwise there is a 100% match rate. The bitbake targets could be
'world core-image-sato-sdk', and target machines could be qemux86_64
and qemuarm64.

Just to be sure that mismatches elsewhere will be reported as CDN
misses, adding my shadow 4.14 update and re-running the command shows:

The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.17.bb:do_package
/srv/work/alex/poky/meta/recipes-sato/pulseaudio-sato/pulseaudio-client-conf-sato_1.bb:do_package
/srv/work/alex/poky/meta/recipes-core/initscripts/initscripts_1.0.bb:do_package
/srv/work/alex/poky/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_package
/srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-ucm-conf_1.2.10.bb:do_package
/srv/work/alex/poky/meta/recipes-kernel/wireless-regdb/wireless-regdb_2023.09.01.bb:do_package
/srv/work/alex/poky/meta/recipes-sato/packagegroups/packagegroup-core-x11-sato.bb:do_package
/srv/work/alex/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_6.5.bb:do_package
/srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-ssh-dropbear.bb:do_package
/srv/work/alex/poky/meta/recipes-graphics/packagegroups/packagegroup-core-x11.bb:do_package
/srv/work/alex/poky/meta/recipes-sato/shutdown-desktop/shutdown-desktop.bb:do_package
virtual:native:/srv/work/alex/poky/meta/recipes-support/libbsd/libbsd_0.11.7.bb:do_prepare_recipe_sysroot
/srv/work/alex/poky/meta/recipes-support/ca-certificates/ca-certificates_20211016.bb:do_package
/srv/work/alex/poky/meta/recipes-core/sysvinit/sysvinit-inittab_2.88dsf.bb:do_package
virtual:native:/srv/work/alex/poky/meta/recipes-extended/shadow/shadow_4.14.0.bb:do_recipe_qa
... (a few more missing do_package objects)
===

2. When to run the test, and against which commit in poky?

RP suggested that the test could run in an antiphase with the nightly
a-quick (i.e. 12 hours after). We can do that for a start, but I'm
(perhaps prematurely) worrying that this will be unstable: either
because the objects from the nightly haven't yet propagated to CDN, or
because master has meanwhile (e.g. in the 12 hours that have passed)
gained new commits without corresponding cache objects. Are those real
concerns?

Thoughts?

Alex


On Fri, 22 Sept 2023 at 19:32, Michael Halstead
 wrote:
>
> Right now we have https://cdn.jsdelivr.net/yocto/sstate set up as a CDN 
> mirroring proxy endpoint. It's fast and completely free to the project. 
> However, it will only work for files smaller than 64MB and will send a 
> redirect to the main server at sstate.yoctoproject.org for files larger than 
> 64MB.
>
> The good news is that we have a fast free CDN provided by 
> https://www.jsdelivr.com/ and their many sponsors for 99.4% of our files.
>
> Here is an estimated distribution of sstate files currently available:
> 1 KB-4 KB: 103168
> 4 KB-16 KB: 1218048
> 16 KB-64 KB: 1886976
> 64 KB-256 KB: 1225216
> 256 KB-1 MB: 170752
> 1 MB-4 MB: 92928
> 4 MB-16 MB: 83200
> 16 MB-64 MB: 40448
> 64 MB-256 MB: 17664
> 256 MB-1 GB: 7936
> 1 GB-4 GB: 2560
>
> I'm still working on a CDN solution for the larger files but we can begin 
> testing https://cdn.jsdelivr.net/yocto/sstate today.
>
> Here is an example link to a file that is currently popular from each 
> location:
>
> http://sstate.yoctoproject.org/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst
>
> https://cdn.jsdelivr.net/yocto/sstate/all/universal/c6/81/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:11:c681f19e2786732c7920860d0b27cdcf3a79c9441d9bdaa82319a12e96e6246e_deploy_source_date_epoch.tar.zst
>
> I'm also attaching a list of the largest files in sstate. Anything we can do 
> to reduce the size of these or to split them into smaller pieces will help 
> with hosting and with the user's download experience.
>
> --
> Michael H

Re: [yocto] CDN for sstate.yoctoproject.org

[Richard Purdie]

On Tue, 2023-10-17 at 17:33 +0200, Alexander Kanavin wrote:
> Thanks for working on this! I finally got to playing with CDN mirror a
> bit, and it seems to basically work ok, so next I'm going to write AB
> tests that check that it remains useful. Specifically:
> 
> 1. What should the test be?
> 
> I tried 'bitbake -S printdiff core-image-sato' on a blank build
> directory, and the result against current master is:
> 
> ===
> Checking sstate mirror object availability: 100%
> > ##|
> Time: 0:05:35
> The differences between the current build and any cached tasks start
> at the following tasks:
> /srv/work/alex/poky/meta/recipes-sato/images/core-image-sato.bb:do_deploy_source_date_epoch
> /srv/work/alex/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_install
> ===
> 
> I think this is pretty good! The two missing objects depend on the
> current date, and so should go to the exception list in the test, and
> otherwise there is a 100% match rate. The bitbake targets could be
> 'world core-image-sato-sdk', and target machines could be qemux86_64
> and qemuarm64.

That match does look good. As you say, the revision of the metadata
will change base-files so that is expected.

I'm torn on the targets to test as sato-sdk is a large image and world
is a lot of work. I'd be tempted to test sato, weston and full-cmdline?
World is a good test I guess and if from sstate, shouldn't have that
much of an issue. It does also prove things are working.

> Just to be sure that mismatches elsewhere will be reported as CDN
> misses, adding my shadow 4.14 update and re-running the command shows:
> 
> The differences between the current build and any cached tasks start
> at the following tasks:
> /srv/work/alex/poky/meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.17.bb:do_package
> /srv/work/alex/poky/meta/recipes-sato/pulseaudio-sato/pulseaudio-client-conf-sato_1.bb:do_package
> /srv/work/alex/poky/meta/recipes-core/initscripts/initscripts_1.0.bb:do_package
> /srv/work/alex/poky/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_package
> /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-ucm-conf_1.2.10.bb:do_package
> /srv/work/alex/poky/meta/recipes-kernel/wireless-regdb/wireless-regdb_2023.09.01.bb:do_package
> /srv/work/alex/poky/meta/recipes-sato/packagegroups/packagegroup-core-x11-sato.bb:do_package
> /srv/work/alex/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_6.5.bb:do_package
> /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-ssh-dropbear.bb:do_package
> /srv/work/alex/poky/meta/recipes-graphics/packagegroups/packagegroup-core-x11.bb:do_package
> /srv/work/alex/poky/meta/recipes-sato/shutdown-desktop/shutdown-desktop.bb:do_package
> virtual:native:/srv/work/alex/poky/meta/recipes-support/libbsd/libbsd_0.11.7.bb:do_prepare_recipe_sysroot
> /srv/work/alex/poky/meta/recipes-support/ca-certificates/ca-certificates_20211016.bb:do_package
> /srv/work/alex/poky/meta/recipes-core/sysvinit/sysvinit-inittab_2.88dsf.bb:do_package
> virtual:native:/srv/work/alex/poky/meta/recipes-extended/shadow/shadow_4.14.0.bb:do_recipe_qa
> ... (a few more missing do_package objects)
> ===
> 
> 2. When to run the test, and against which commit in poky?
> 
> RP suggested that the test could run in an antiphase with the nightly
> a-quick (i.e. 12 hours after). We can do that for a start, but I'm
> (perhaps prematurely) worrying that this will be unstable: either
> because the objects from the nightly haven't yet propagated to CDN, or
> because master has meanwhile (e.g. in the 12 hours that have passed)
> gained new commits without corresponding cache objects. Are those real
> concerns?
> 
> Thoughts?

You're right, we need to run the test against a commit we know has been
built on the autobuilder. If we run with a newer commit we could easily
see mismatch since it won't have been built yet.

I'm less worried about CDN propagation, that should happen quickly and
if something isn't present, it might just be slow to obtain/lookup as
it ripples through the system.

Either we start logging what we've built so we get the last known
revisions, or we run the test as part of a-quick/a-full at the end of
the build? I don't really want to extend the build but I'm not sure we
may have much choice.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61387): https://lists.yoctoproject.org/g/yocto/message/61387
Mute This Topic: https://lists.yoctoproject.org/mt/101525879/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: 
https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] CDN for sstate.yoctoproject.org

[Alexander Kanavin]

On Wed, 18 Oct 2023 at 08:45, Richard Purdie
 wrote:
> I'm torn on the targets to test as sato-sdk is a large image and world
> is a lot of work. I'd be tempted to test sato, weston and full-cmdline?
> World is a good test I guess and if from sstate, shouldn't have that
> much of an issue. It does also prove things are working.

I ran '-S printdiff world' on a blank build directory. First,
scalability isn't great:

Initialising tasks: 100%
|##|
Time: 0:24:19
Checking sstate mirror object availability: 100%
|##|
Time: 0:12:14

So it's taking 36 minutes just preparing to fetch the objects, and 2/3
of that time goes into communicating with hash equivalence server
(e.g. BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687").

Second, there are significant misses. I don't have a clear theory
where they come from, just want to list them:

The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-core/meta/meta-ide-support.bb:do_populate_ide_support
/srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-synaptics_1.9.2.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-devtools/bootchart2/bootchart2_0.14.9.bb:do_create_runtime_spdx
/srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-devtools/python/python3-dbusmock_0.29.1.bb:do_create_runtime_spdx
/srv/work/alex/poky/meta/recipes-devtools/dpkg/dpkg_1.22.0.bb:do_configure
/srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-vmmouse_13.2.0.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-graphics/glew/glew_2.2.0.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-evdev_2.10.6.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-graphics/libva/libva_2.19.0.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-sato/webkit/libwpe_1.14.1.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-meta-base.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-python_1.22.6.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-extended/cups/cups_2.4.6.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-gnome/libdazzle/libdazzle_3.44.0.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_package
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-devtools_1.22.6.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:do_package
/srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-mouse_1.9.5.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-graphics/waffle/waffle_1.7.2.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-tools_1.2.5.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-rtsp-server_1.22.6.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-devtools/apt/apt_2.6.1.bb:do_install
/srv/work/alex/poky/meta/recipes-gnome/gtk+/gtk4_4.12.3.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-devtools/devel-config/distcc-config.bb:do_create_runtime_spdx
/srv/work/alex/poky/meta/recipes-gnome/libhandy/libhandy_1.8.2.bb:do_collect_spdx_deps
/srv/work/alex/poky/meta/recipes-support/vim/vim_9.0.bb:do_collect_spdx_deps

So I think we should start with specific images first - minimal,
full-cmdline, weston, sato and sato-sdk are all much faster to check.

On qemux86_64 none of them show misses, but on qemuarm64 there are
problems with sato, sato-sdk and weston, i.e. sato-sdk shows:

The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.42.10.bb:do_package_write_rpm
/srv/work/alex/poky/meta/recipes-connectivity/connman/connman-gnome_0.7.bb:do_package_write_rpm
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-examples_1.18.6.bb:do_package_write_rpm
/srv/work/alex/poky/meta/recipes-sato/images/core-image-sato-sdk.bb:do_deploy_source_date_epoch
/srv/work/alex/poky/meta/recipes-graphics/xorg-font/font-util_1.4.1.bb:do_package_write_rpm
/srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:do_package
/srv/work/alex/poky/meta/recipes-gnome/librsvg/librsvg_2.56.3.bb:do_package_write_rpm
/srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-bad_1.22.6.bb:do_package_write_rpm


I'm not sure if that's because cache for the current master needs to
be populated properly first, or if there's a deeper issue.

> Either we start logging what we've bu

Re: [yocto] CDN for sstate.yoctoproject.org

[Richard Purdie]

On Wed, 2023-10-18 at 14:02 +0200, Alexander Kanavin wrote:
> On Wed, 18 Oct 2023 at 08:45, Richard Purdie
>  wrote:
> > I'm torn on the targets to test as sato-sdk is a large image and world
> > is a lot of work. I'd be tempted to test sato, weston and full-cmdline?
> > World is a good test I guess and if from sstate, shouldn't have that
> > much of an issue. It does also prove things are working.
> 
> I ran '-S printdiff world' on a blank build directory. First,
> scalability isn't great:
> 
> Initialising tasks: 100%
> > ##|
> Time: 0:24:19
> Checking sstate mirror object availability: 100%
> > ##|
> Time: 0:12:14
> 
> So it's taking 36 minutes just preparing to fetch the objects, and 2/3
> of that time goes into communicating with hash equivalence server
> (e.g. BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687").

FWIW the second time an artefact is pulled from the CDN, it will be
much faster. I don't know how much that is a factor here.

We know hashequiv is slow, particularly from europe. We've ideas on
that we're looking into but for now it is what it is. Part of this work
was to find out what the issues are so we write that one up.

> Second, there are significant misses. I don't have a clear theory
> where they come from, just want to list them:
> 
> The differences between the current build and any cached tasks start
> at the following tasks:
> /srv/work/alex/poky/meta/recipes-core/meta/meta-ide-support.bb:do_populate_ide_support
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-synaptics_1.9.2.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-devtools/bootchart2/bootchart2_0.14.9.bb:do_create_runtime_spdx
> /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-devtools/python/python3-dbusmock_0.29.1.bb:do_create_runtime_spdx
> /srv/work/alex/poky/meta/recipes-devtools/dpkg/dpkg_1.22.0.bb:do_configure
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-vmmouse_13.2.0.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-graphics/glew/glew_2.2.0.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-evdev_2.10.6.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-graphics/libva/libva_2.19.0.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-sato/webkit/libwpe_1.14.1.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-meta-base.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-python_1.22.6.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-extended/cups/cups_2.4.6.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-gnome/libdazzle/libdazzle_3.44.0.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:do_package
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-devtools_1.22.6.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:do_package
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-mouse_1.9.5.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-graphics/waffle/waffle_1.7.2.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-tools_1.2.5.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-rtsp-server_1.22.6.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-devtools/apt/apt_2.6.1.bb:do_install
> /srv/work/alex/poky/meta/recipes-gnome/gtk+/gtk4_4.12.3.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-devtools/devel-config/distcc-config.bb:do_create_runtime_spdx
> /srv/work/alex/poky/meta/recipes-gnome/libhandy/libhandy_1.8.2.bb:do_collect_spdx_deps
> /srv/work/alex/poky/meta/recipes-support/vim/vim_9.0.bb:do_collect_spdx_deps

We'll have to dig into this.

> So I think we should start with specific images first - minimal,
> full-cmdline, weston, sato and sato-sdk are all much faster to check.

Agreed, lets start with simpler images, full-cmdline, minimal, sato and
weston. We can increase coverage as we move forward (world, sato-sdk).

> On qemux86_64 none of them show misses, but on qemuarm64 there are
> problems with sato, sato-sdk and weston, i.e. sato-sdk shows:
> 
> The differences between the current build and any cached tasks start
> at the following tasks:
> /srv/work/alex/poky/meta/recipes-gnome/gdk-pixbuf/gdk-pixbuf_2.42.10.bb:do_package_write_rpm
> /srv/work/alex/poky/meta/recipes-connectivity/connman/connman-gnome_0.7.bb:do_package_write_rpm
> /srv/work/alex/poky/meta/recipes-multimedia/gstream

Re: [yocto] CDN for sstate.yoctoproject.org

[Michael Halstead]

Some CDN stats are now published at https://www.jsdelivr.com/oss-cdn/yocto.
Let me know if you notice anything that needs to be changed.

On Wed, Oct 18, 2023 at 10:33 AM Richard Purdie <
richard.pur...@linuxfoundation.org> wrote:

> On Wed, 2023-10-18 at 14:02 +0200, Alexander Kanavin wrote:
> > On Wed, 18 Oct 2023 at 08:45, Richard Purdie
> >  wrote:
> > > I'm torn on the targets to test as sato-sdk is a large image and world
> > > is a lot of work. I'd be tempted to test sato, weston and full-cmdline?
> > > World is a good test I guess and if from sstate, shouldn't have that
> > > much of an issue. It does also prove things are working.
> >
> > I ran '-S printdiff world' on a blank build directory. First,
> > scalability isn't great:
> >
> > Initialising tasks: 100%
> > >
> ##|
> > Time: 0:24:19
> > Checking sstate mirror object availability: 100%
> > >
> ##|
> > Time: 0:12:14
> >
> > So it's taking 36 minutes just preparing to fetch the objects, and 2/3
> > of that time goes into communicating with hash equivalence server
> > (e.g. BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687").
>
> FWIW the second time an artefact is pulled from the CDN, it will be
> much faster. I don't know how much that is a factor here.
>
> We know hashequiv is slow, particularly from europe. We've ideas on
> that we're looking into but for now it is what it is. Part of this work
> was to find out what the issues are so we write that one up.
>
> > Second, there are significant misses. I don't have a clear theory
> > where they come from, just want to list them:
> >
> > The differences between the current build and any cached tasks start
> > at the following tasks:
> > /srv/work/alex/poky/meta/recipes-core/meta/meta-ide-support.bb:
> do_populate_ide_support
> > /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/
> xf86-input-synaptics_1.9.2.bb:do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-devtools/bootchart2/bootchart2_0.14.9.bb:
> do_create_runtime_spdx
> >
> /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-devtools/python/python3-dbusmock_0.29.1.bb:
> do_create_runtime_spdx
> > /srv/work/alex/poky/meta/recipes-devtools/dpkg/dpkg_1.22.0.bb:
> do_configure
> >
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-vmmouse_13.2.0.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-graphics/glew/glew_2.2.0.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-evdev_2.10.6.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-graphics/libva/libva_2.19.0.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-sato/webkit/libwpe_1.14.1.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-meta-base.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/
> gstreamer1.0-python_1.22.6.bb:do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-extended/cups/cups_2.4.6.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-gnome/libdazzle/libdazzle_3.44.0.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-graphics/igt-gpu-tools/igt-gpu-tools_git.bb:
> do_package
> >
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gst-devtools_1.22.6.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-core/packagegroups/packagegroup-core-sdk.bb:
> do_package
> >
> /srv/work/alex/poky/meta/recipes-graphics/xorg-driver/xf86-input-mouse_1.9.5.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-graphics/waffle/waffle_1.7.2.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-multimedia/alsa/alsa-tools_1.2.5.bb:
> do_collect_spdx_deps
> >
> /srv/work/alex/poky/meta/recipes-multimedia/gstreamer/gstreamer1.0-rtsp-server_1.22.6.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-devtools/apt/apt_2.6.1.bb:do_install
> > /srv/work/alex/poky/meta/recipes-gnome/gtk+/gtk4_4.12.3.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-devtools/devel-config/distcc-config.bb:
> do_create_runtime_spdx
> > /srv/work/alex/poky/meta/recipes-gnome/libhandy/libhandy_1.8.2.bb:
> do_collect_spdx_deps
> > /srv/work/alex/poky/meta/recipes-support/vim/vim_9.0.bb:
> do_collect_spdx_deps
>
> We'll have to dig into this.
>
> > So I think we should start with specific images first - minimal,
> > full-cmdline, weston, sato and sato-sdk are all much faster to check.
>
> Agreed, lets start with simpler images, full-cmdline, minimal, sato and
> weston. We can increase coverage as we move forward (world, sato-sdk).
>
> > On qemux86