Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
> On Thu, 14 Sept 2023 at 21:54, Richard Purdie > wrote: > > The tools are already supposed to support doing this with local file > > sstate sources, they just do a bad job at getting the diffs right. One > > intent of this work item was to try and understand why they don't work > > and address that so at least for filesystem sstate mirrors, you can > > get better results. I don't know how we solve the remote http issue as yet. > > I ran a few experiments with bitbake -S printdiff, and so far I haven't yet > seen > an unhelpful result. Here's the most impressive one: > > $ bitbake libsolv-native (uses cmake-native) $ git revert version > update> $ rm -rf build/tmp/ $ bitbake -S printdiff libsolv-native ... > The differences between the current build and any cached tasks start at the > following tasks: > /srv/work/alex/poky/meta/recipes-devtools/cmake/cmake- > native_3.27.4.bb:do_recipe_qa > NOTE: Writing task signature files > Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc > > Task cmake-native:do_recipe_qa couldn't be used from the cache because: > We need hash > 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb, > closest matching task was > 85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7 > basehash changed from > 7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to > 7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09 > Variable PV value changed from '3.27.5' to '3.27.4' > Variable SRC_URI[sha256sum] value changed from > '5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' > to > '0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af' > > That's pretty good, isn't it? It does print both what needs to be re-run, and > *why* as well. > > I have no idea yet what kind of magic it does to find the 'closest matching > task' > in sstate, but if this breaks down in some other scenarios, we need to find > them > to get a starting point for making the tools better. Ideas? I'm ready to try > them > :) That is very impressive and I'd also love to hear about what heuristics it uses. We use Artifactory to host our sstate. (Artifactory doesn’t have specific support for it, it's just basically acting as a generic HTTP server. But it makes things like LDAP easy). I have for a while been thinking of building a tool that tries to find the "closest" sstate on the server and then recursively runs bitbake-diffsig on it. The tool was going to be called 'why-not-sstate'. Thanks, Chris -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61069): https://lists.yoctoproject.org/g/yocto/message/61069 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 21 Sept 2023 at 16:39, chris.lapla...@agilent.com wrote: > That is very impressive and I'd also love to hear about what heuristics it > uses. It's actually rather simple. It uses glob.glob on stamps in tmp/, then on local sstate to find possible matches, then sorts them by mtime and takes the most recent. It's what would work most of the time, but we could add printdiff-all (print difference with all sstate matches) or printdiff-N (N most recent). It also could abstain from dumping locked-sigs.inc into cwd with both -S none and -S printdiff, unless explicitly asked I just discovered there's also scripts/bitbake-whatchanged (that hasn't seen activity in years and is neither documented nor tested). Unsurprisingly then, it doesn't work in the same scenario: alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged libsolv-native Figuring out the STAMPS_DIR ... Generating the new stamps ... (need several minutes) === Summary: (0 changed, 0 unchanged) Newly added: 0 PV changed: 0 PR changed: 0 Dependencies changed: 0 Removing the newly generated stamps dir ... Maybe this is what RP was referring to when he said the tools don't work properly? Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61077): https://lists.yoctoproject.org/g/yocto/message/61077 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Fri, 2023-09-22 at 11:17 +0200, Alexander Kanavin wrote: > On Thu, 21 Sept 2023 at 16:39, chris.lapla...@agilent.com > wrote: > > > That is very impressive and I'd also love to hear about what heuristics it > > uses. > > It's actually rather simple. It uses glob.glob on stamps in tmp/, then > on local sstate to find possible matches, then sorts them by mtime and > takes the most recent. It's what would work most of the time, but we > could add printdiff-all (print difference with all sstate matches) or > printdiff-N (N most recent). It also could abstain from dumping > locked-sigs.inc into cwd with both -S none and -S printdiff, unless > explicitly asked > > I just discovered there's also scripts/bitbake-whatchanged (that > hasn't seen activity in years and is neither documented nor tested). > Unsurprisingly then, it doesn't work in the same scenario: > > > alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged > libsolv-native > Figuring out the STAMPS_DIR ... > Generating the new stamps ... (need several minutes) > > === Summary: (0 changed, 0 unchanged) > Newly added: 0 > PV changed: 0 > PR changed: 0 > Dependencies changed: 0 > > Removing the newly generated stamps dir ... > > > Maybe this is what RP was referring to when he said the tools don't > work properly? No, I've believed that should probably be removed. I think there was a recent change to it. I think we had a major step change in this functionality working when this was fixed: https://git.yoctoproject.org/poky/commit/?id=84a7485025dd4473403b8da36a0c979a3afd5e93 and this test case was added: https://git.yoctoproject.org/poky/commit/?id=1bdcd76d2968c3cc6ec2815afceba1cf98efd6d5 Things which used to be problematic: a) changes involving changes to gcc-source since it uses a shared sources stamps which confused the tools (at least used to). That may have been before gcc-source became a recipe? b) changes to a very common component (e.g. autoconf-native's do_configure) which make it hard to understand where the root cause of the changes came from c) changes which affect many recipes at once, e.g. the do_configure function in base.bbclass It might be helpful to write test cases for the scenario you showed as working above and some of the ones I mention above, then we can document they work and have an easier way to add tests for issues if/as/when we identify the problematic scenarios in future. As you mention, it also uses mtime so perhaps issues happen if you run a different build, then try and go back to the other config? I suspect once you understand the algorithm the code uses, you can pick holes in it. Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61078): https://lists.yoctoproject.org/g/yocto/message/61078 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Fri, 22 Sept 2023 at 12:42, Richard Purdie wrote: > Things which used to be problematic: > > a) changes involving changes to gcc-source since it uses a shared > sources stamps which confused the tools (at least used to). That may > have been before gcc-source became a recipe? > b) changes to a very common component (e.g. autoconf-native's > do_configure) which make it hard to understand where the root cause of > the changes came from > c) changes which affect many recipes at once, e.g. the do_configure > function in base.bbclass > > It might be helpful to write test cases for the scenario you showed as > working above and some of the ones I mention above, then we can > document they work and have an easier way to add tests for issues > if/as/when we identify the problematic scenarios in future. I've now written down the tests for these three scenarios and got them to pass (in oe-selftest too \0/): https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all (check the commit message too) I am going to look closer at bitbake-whatchanged, what it aims to do and why it doesn't work. I have a hunch it can produce useful high level reports, and so shouldn't be simply thrown away. 'bitbake -S printdiff' is too techy and verbose for some use cases. Maybe we can fold that functionality into 'bitbake -S whatchanged'. I'm on holiday next week. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61156): https://lists.yoctoproject.org/g/yocto/message/61156 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 2023-09-28 at 18:43 +0200, Alexander Kanavin wrote: > On Fri, 22 Sept 2023 at 12:42, Richard Purdie > wrote: > > > Things which used to be problematic: > > > > a) changes involving changes to gcc-source since it uses a shared > > sources stamps which confused the tools (at least used to). That may > > have been before gcc-source became a recipe? > > b) changes to a very common component (e.g. autoconf-native's > > do_configure) which make it hard to understand where the root cause of > > the changes came from > > c) changes which affect many recipes at once, e.g. the do_configure > > function in base.bbclass > > > > It might be helpful to write test cases for the scenario you showed as > > working above and some of the ones I mention above, then we can > > document they work and have an easier way to add tests for issues > > if/as/when we identify the problematic scenarios in future. > > I've now written down the tests for these three scenarios and got them > to pass (in oe-selftest too \0/): > https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all > (check the commit message too) > > I am going to look closer at bitbake-whatchanged, what it aims to do > and why it doesn't work. I have a hunch it can produce useful high > level reports, and so shouldn't be simply thrown away. 'bitbake -S > printdiff' is too techy and verbose for some use cases. Maybe we can > fold that functionality into 'bitbake -S whatchanged'. I've wondered if we should split bitbake -S printdiff into a separate utility? It exists from a time before we had bitbake command APIs. I'm curious to see what you find with analysis of bitbake-whatchanged. I'm also somewhat surprised the scenarios you're testing all work! I'm guess one of the commits I pointed to must have fixed them (the removal of paths from the sig files)? Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61157): https://lists.yoctoproject.org/g/yocto/message/61157 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 28 Sept 2023 at 18:49, Richard Purdie wrote: > I've wondered if we should split bitbake -S printdiff into a separate > utility? It exists from a time before we had bitbake command APIs. It can also be a simple shell wrapper. I've separated -S lockedsigs into it's own option as well, so it can be explicitly requested (just need to fix up a couple of selftests that relied on -S none to get locked-sigs.inc): https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all&id=41e8a8a79bac4e3b9eb74a99e32e0d26ac8af0c5 > I'm curious to see what you find with analysis of bitbake-whatchanged. > I'm also somewhat surprised the scenarios you're testing all work! It's not 100% perfect. One out of 2*3=6 scenarios isn't fully functional (gcc-source:do_preconfigure signature isn't found from sstate even though it exists there - but it is found if it's in tmp/stamps/). I left a FIXME in the test. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61158): https://lists.yoctoproject.org/g/yocto/message/61158 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Thu, 28 Sept 2023 at 18:49, Richard Purdie
wrote:
> I'm curious to see what you find with analysis of bitbake-whatchanged.
I've taken a look a the script. It obtains the current location of
STAMPS_DIR, then runs this:
# Generate the new stamps dir
print("Generating the new stamps ... (need several minutes)")
cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
args.recipe)
Then it walks both trees, matching up file names with a regex:
# Match the stamp's filename
# group(1): PE_PV (may no PE)
# group(2): PR
# group(3): TASK
# group(4): HASH
stamp_re =
re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)")
Then there's some code that finds out what changed in the above
between the two sets.
I don't see a way to make it work: messing about with STAMPS_DIR like
that isn't supported, and will either do nothing, or remove the
original stamps. Also stamp filenames aren't really a 'public API',
are they?
Should the script simply be removed, or is there some better way to
re-implement answering the 'what has changed' question in a way that
doesn't flood the console with task hashes? I'd be glad to get
suggestions for this.
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61162): https://lists.yoctoproject.org/g/yocto/message/61162
Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Fri, 2023-09-29 at 14:06 +0200, Alexander Kanavin wrote:
> On Thu, 28 Sept 2023 at 18:49, Richard Purdie
> wrote:
>
> > I'm curious to see what you find with analysis of bitbake-whatchanged.
>
> I've taken a look a the script. It obtains the current location of
> STAMPS_DIR, then runs this:
>
> # Generate the new stamps dir
> print("Generating the new stamps ... (need several minutes)")
> cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
> args.recipe)
>
> Then it walks both trees, matching up file names with a regex:
>
> # Match the stamp's filename
> # group(1): PE_PV (may no PE)
> # group(2): PR
> # group(3): TASK
> # group(4): HASH
> stamp_re =
> re.compile("(?P.*)-(?Pr\d+)\.(?Pdo_\w+)\.(?P[^\.]*)")
>
> Then there's some code that finds out what changed in the above
> between the two sets.
>
> I don't see a way to make it work: messing about with STAMPS_DIR like
> that isn't supported, and will either do nothing, or remove the
> original stamps. Also stamp filenames aren't really a 'public API',
> are they?
>
> Should the script simply be removed, or is there some better way to
> re-implement answering the 'what has changed' question in a way that
> doesn't flood the console with task hashes? I'd be glad to get
> suggestions for this.
I'd prefer to see some dedicated bitbake API used even if we need to
create/add it. tinfoil and some of the bblock/unlock work shows we can
get stamp data, the question would be how to get it without
"disturbing" the existing build.
By using dedicated API, we'd be able to control the console output.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61163): https://lists.yoctoproject.org/g/yocto/message/61163
Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe:
https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?
On Fri, 29 Sept 2023 at 14:27, Richard Purdie wrote: > I'd prefer to see some dedicated bitbake API used even if we need to > create/add it. tinfoil and some of the bblock/unlock work shows we can > get stamp data, the question would be how to get it without > "disturbing" the existing build. > > By using dedicated API, we'd be able to control the console output. I just noticed that compare_sigfiles() has a 'collapsed' argument, which, when set to True, omits most or all of the task hash printing, and with that its output should approximate what bitbake-whatchanged is aiming to do. This is currently used only by buildhistory, but it could be used by -S printdiff too, e.g. by having verbose and concise modes. I'll run some experiments, and let's see what the overall output looks like in real scenarios (e.g. 4.3_M3 vs current master). Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#61165): https://lists.yoctoproject.org/g/yocto/message/61165 Mute This Topic: https://lists.yoctoproject.org/mt/101503345/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
