On Sat, 2023-11-04 at 11:29 +0100, adrian.freiho...@gmail.com wrote:
> Hi Alex, hi Richard
>
> After some internal discussions, I would like to clarify my previous
> answers on this topic.
>
> * Usually there are two different workflows
> - application developers: could use an SDK with a locked sstate-cache.
> - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
> * A locked SDK
> - can work with setscene from SSTATE_MIRRORS
> - setscene does caching in the SSTATE_DIR (no issue about that)
> - But network problems can occur during the initial build because
> bitbake executes many independent setscene tasks. Opening so many
> independent connections slows down the build, especially if the
> server treats them as a denial of service attack.
> - The denial of service problem is difficult to solve because each
> setscene task runs in its own bibtake task. Reusing a connection to
> download multiple sstate artifacts seems almost impossible.
> This is much easier to solve with separate sstate download script.
FWIW, we did have a similar issue with do_fetch overloading
servers/proxies/ISPs and added:
do_fetch[number_threads] = "4"
Finding the right place to put a thread limit on overall setscene tasks
is harder but in theory possible. Or perhaps a "network capable tasks"
thread limit?
Is the overload caused by the initial query of sstate presence, or,
does it happen when the setscene tasks themselves run?
> * An unlocked SDK
> - Tries to download the sstate cache for changed recipes and their
> dependencies, which obviously can't work.
> - The useless download requests slow down the build considerably and
> cause a high load on the servers without any benefit.
Is this sstate over http(s) or something else? I seem to remember you
mentioning sftp. If this were using sftp, it would be horribly slow as
it was designed for a light overhead "does this exist?" check which
http(s) can manage well.
Recently we've been wondering about teaching the hashequiv server about
"presence", which would then mean the build would only query things
that stood a good chance of existing.
> - A script which gets a list of sstate artifacts from bitbake and then
> does a upfront download works much better
> + The script runs only when the user calls it or the SDK gets boot-
> strapped
> + The script uses a reasonable amount of parallel connections which
> are re-used for more then one artifact download
Explaining to users they need to do X before Y quickly gets tiring,
both for people explaining it and the people doing it trying to
remember. I'd really like to get to a point where the system "does the
right thing" if we can.
I don't believe the problems you describe are insurmountable. If you
are using sftp, that is going to be a big chunk of the problem as the
system assumes something faster is available. Yes, I've taken patches
to make sftp work but it isn't recommended at all. I appreciate there
would be reasons why you use sftp but if it is possible to get a list
of "available sstate" via other means, it would improve things.
> * Idea for a smart lock/unlock implementation
> - Form a user's perspective a locked vs. an unlocked SDK does not make
> much sense. It makes more sense if the SDK would automatically
> download the sstate-cache if it is expected to be available.
> Lets think about an implementation (which allows to override the
> logic) to switch from automatic to manual mode:
>
> SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
What determines this availability? I worry that is something very
fragile and specific to your use case. It is also not an all or nothing
binary thing.
> In our case the sstate mirror is expected to provide all artifacts
> for tagged commits and for some git branches of the layer
> repositories.
> The sstate is obviousely not usable for a "dirty" git layer
> repository.
That isn't correct and isn't going to work. If I make a single change
locally, there is a good chance that 99.9% of the sstate could still be
valid in some cases. Forcing the user through 10 hours of rebuild when
potentially that much was available is a really really bad user
experience.
> That's what the is_sstate_mirror_available function
> could check to automatically enable and disable lazy downloads.
>
> - If is_sstate_mirror_available() returns false, it should still be
> possible to initiate a sstate-cache download manually.
>
> * Terminology
> - Older Yocto Releases:
> + eSDK means an installer which provides a different environment with
> different tools
> + The eSDK was static, with a locked sstate cache
> + Was for one MACHINE, for one image...
> - Newer Yocto Releases:
> + The bitbake environment offers all features of the eSDK installer. I
> consider this as already implemented with meta-ide-support and
> build-sysroots.
Remember bblock and bbunlock too. These provide a way to fix or unlock
specific sections of the codebase. Usually a developer has a pretty
good idea of which bits they want to allow to change. I don't think
people have yet realised/explored the potential these offer.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61611): https://lists.yoctoproject.org/g/yocto/message/61611
Mute This Topic: https://lists.yoctoproject.org/mt/102320109/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe:
https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
