Hi Richard (and all), @Richard: Thanks for the input regarding the topic and apologies for not replying until now, I was a bit buried under workload and simultaneously completing my Masters studies. I also wanted to take some time to digest your comments amid working on this CDEPENDS concept, and I just want to say that I'm very open to finding another way to implement this concept of rebuild avoidance in Yocto.
I think the idea of a non-invasive approach can be a bit tricky, but your idea of a secondary sstate signature and mapping table sounds workable to me. (In fact, as I slowly developed out this concept, I think it's almost required). One point though with the removal of the metadata changes is that the view of "what is compatible" will be concrete and cannot differ among children. Sometimes you have only specific children of a recipe that can handle compatibility changes (and at different levels of compatibility) while others cannot handle any change at all, and that is why I developed with the metadata changes as a basis. But definitely, yes, the change to metadata makes things more complex and does in a sense impose policy, which I understand would be undesirable. Maybe we can figure out an approach that avoids metadata changes but keeps this ability to differ compatibility handling among children. Currently I have a working proof of concept (I've developed out the earlier shared patches further) that I'd like to share soon and demonstrate. It still needs some further refinement but I think it's showing some interesting results already. I'll think further about the topic and see what I can come up with that matches your line of thoughts. @Martin Jansa: Thank you for the comments and information, it's good to know that there's some interest from the community to look into this idea. @Mike Looijmans: Yes, that's essentially the problem I want to resolve with this proof of concept, breaking long build chains at points where a clear precision cut could've been made depending on whether a rebuild is actually required or not (i.e. the bitstream tool not changing). In that case I would even use the tool version and assume even if the tool binary has changed, that the output result of tool would stay the same so it is compatible in essence (and avoid rebuilding 31 hours of stuff). @Trevor Woerner: Thanks for the suggestion, I was planning to cross-post when I had a more stable proof of concept code to share. Note: I'll be following my colleague, Mario Domenech Goulart (mario-goulart), to OEDEM 2017 where I hope to discuss the current concept code and show some potential use cases. Kind regards, Michael Ho -- BMW Car IT GmbH Michael Ho Spezialist Entwicklung - Linux Software Integration Lise-Meitner-Str. 14 89081 Ulm Tel.: +49 731 3780 4071 Mobil: +49 152 5498 0471 Fax: +49-731-37804-001 Mail: michael...@bmw-carit.de Web: http://www.bmw-carit.de -------------------------------------------------------- BMW Car IT GmbH Gechäftsführer: Kai-Uwe Balszuweit und Alexis Trolin Sitz und Registergericht: München HRB 134810 -------------------------------------------------------- ________________________________________ From: Richard Purdie <richard.pur...@linuxfoundation.org> Sent: 21 September 2017 18:11 To: Michael Ho; yocto@yoctoproject.org Subject: Re: [yocto] RFC: Backwards compatibility checking sstate-cache On Fri, 2017-06-30 at 09:48 +0000, Michael Ho wrote: > Hi, at BMW Car IT we are working on an experimental feature to > improve sstate cache hits and we are looking for comments on the > approach who might have some insights to the problem and seeing if > anyone is interested in the feature for mainline. Sorry I didn't see this before now but it was brought to my attention. I have given this problem quite some thought off and on. I do worry that the interface you propose is complex and requires changes throughout the metadata and those changes impose "policy". One of our strengths is that we're managed to keep "policy" out of the metadata. In this context by policy, I mean when a recipe is equivalent and when it is not. I do have a counter-proposal of how we could solve this in a less invasive way. This isn't implemented but wouldn't in theory be hard to do. The idea would be to have some kind of equivalence list. The first built object's sstate checksum would become the definitive checksum and the object added to the sstate cache. If a new object is built, it would be compared with the one in sstate. If deemed equivalent (by whatever policy), a mapping would be added to the list. If not, there is no mapping and it becomes a new definitive checksum. All runqueue would then have to do is present its list of sstate checksums to the list and convert them (in dependency order) into canonical checksums. This list could be a local equivalence file, or an equivalence server which could resolve list of checksums. Doing it this way totally isolates the comparison from the metadata itself and in my view protects us from future changes in definition of equivalence. How does that sound? Cheers, Richard -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto