On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote: > On Tue, 12 Sept 2023 at 16:44, Stephen Jolley <sjolley.yp...@gmail.com> wrote: > > Alexander Kanavin will be working on the core workflow topic > > I am now ready to start doing this, but before I do, I'd like to > decompose the subject into manageable tasks with a bit of help from RP > and the community: > > https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/ > > ==== > Core Workflow – Process Improvements > > Background > > The project builds everything from source by default. This means it > has a reputation for being slow and heavy. There are ways the project > can accelerate this which means faster workflows and improved > developer experience but these are currently cumbersome to use. > > Rationale > > The project aims to defragment customised embedded Linux. This is > important as if we succeed at this, it gives benefits to the wider > ecosystem through making it easier to inject security fixes and an > ability to share and collaborate without re-inventing the wheel. > > To do this, we need to provide best-in-class support and compete > against binary distributions for usability and speed. One way we can > do this is provide better support for binary artifacts via our sstate > mechanism. We do already have some of this functionality in our > “extensible SDK” or “eSDK”. > > Deliverables > > Enable a public sstate mirror via a content delivery network (CDN) and > populate using the autobuilder > Ensure CDN sstate is reused under normal use case scenarios, > particularly for slow components like rust-native. Identify any common > sstate mismatch causes. Ensure test cases are added to cover the use > cases and prevent regressions. > Add lock and unlock commands to allow specific components to be locked > down to specific sstate checksums or allow them to vary > Allow switching between eSDK and non-eSDK modes > Add tooling so we can understand why something is rebuilding when it > isn’t expected to. > ======= > > So: where to start? Do we need to 'design' something, or maybe time > should go directly into addressing specific sore points? All feedback > welcome. > > There's also an unfinished patchset for adding bblock/bbunlock, which > I would prefer to *not* 'take over and finish' but rather see the > author get it merged: > https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276
To start with I'll try and write down and give a random walk through my thoughts. Certainly we need to get something like that patchset over the line. I think it was blocked on a reply to this email: https://lists.openembedded.org/g/openembedded-core/message/186497 which has suffered whilst I was pulled into the qemuppc mess. I feel bad for not getting to a reply to that. There are design elements to this work. We need to work out how we can make eSDK and "normal" builds more similar and less of an overhead to switch between one and the other. A "bblock all" command does partly get you to an eSDK effectively so part of this may be switching eSDK to use the new lock command. What other differences are there? What other differences are necessary or make sense for the use cases eSDK was designed for? How would you turn an existing build into an eSDK like one? Could you provide a copy of a local build to someone else easily using something like eSDK's tooling? What does the eSDK look like at the end of this. One section we don't have good answers to yet is setup and configuration although I know you've started on some of that. For the task signatures, we need to think about some questions. If I make a change locally, can I query how much will rebuild and how much will be reused? There is bitbake --dry-run but perhaps it is time for a an option (or dedicated separate command?) to give some statistics about what bitbake would do? How much sstate would be reused? That then logically leads into the questions, can we tell what has changed? Why isn't my sstate being reused? For that we perhaps should define some existing scenarios where it is currently very difficult to work this out and then work out how we can report that information to the user. These could become test cases? One of the big problems in the past was that we lost much of the hash information after parsing completed. This meant that if the hashes then didn't match, we couldn't tell why as the original computation was lost. I did some work on allowing us to retain more of the information so that we didn't have to recompute it every time to be able to do processing with it. I have to admit I've totally lost track of where I got to with that. Michael Halstead will be working on setting the CDN up so I'll let him comment on when we'll have things ready for testing with that. We do already have sstate shared from the autobuilder so some basic tests to make sure our "base" shared files do work as expected is something which can happen there already. Another interesting question - would the project be interested in shipping "locked" hashes for a minimal subset of recipes such as rust- native and maybe the cross compilers to bootstrap and speed people's builds, assuming they have a fast network? What does the size of those pieces look like and would it be useful/effective? Probably more questions than answers here but it hopefully gives a bit more insight into some of the directions I'm thinking about. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61006): https://lists.yoctoproject.org/g/yocto/message/61006 Mute This Topic: https://lists.yoctoproject.org/mt/101356418/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
