On Tue, 16 Jan 2024, Richard Purdie wrote:
> On Tue, 2024-01-16 at 09:23 -0500, Robert P. J. Day wrote:
> > a very weird question, to be sure, but i literally just ran across a
> > recipe that does the following (i will paraphrase some stuff to
> > protect the innocent).
> >
> > the recipe processes a bunch of XML source files, and generates from
> > them JSON files, so any change in any of the XML source files should
> > (obviously) trigger a rebuild of the recipe. and here's the problem.
> >
> > the XML files are not copied into the recipe source directory, and
> > are not even referenced by SRC_URI. rather, the files live in a
> > well-known location on the build host (placed there by a monstrous
> > "repo sync"), and to save time and space, the recipe just *symlinks*
> > from inside ${S} to the XML source files outside of ${S}, then
> > proceeds to process them to generate the subsequent JSON files.
> >
> > the complaint is that, when the build pipeline started taking
> > advantage of sstate-cache, even after some of the XML files were
> > changed, the recipe would not rebuild.
> >
> > once i understood what was happening, my immediate reaction was
> > along the lines, "well, since what is under ${S} is simply symlinks to
> > files *outside* of ${S} and (worse) those symlink names will never
> > change (all that will change is the XML files at the other end), how
> > is bitbake supposed to have any idea that those XML files have
> > changed to kick off a rebuild?"
> >
> > so my initial response was, "this does not surprise me at all," but
> > does my reasoning hold up? obviously, this should probably use a
> > combination of bin_package and externalsrc, but as it is now, is it
> > moderately accurate to suggest that it is those symlinks that are
> > preventing bitbake from understanding that anything has changed, so
> > that it continues to use the old sstate-cache?
>
> Inputs to a recipe need to come through SRC_URI. That way bitbake can
> know when they change. If you point at some random place via a symlink
> how is bitbake supposed to know if/when they change?
>
> Yes, pointing at a large tree of files will take time to hash and yes,
> that can make parsing slow. Either you care when it changes or you
> don't.
>
> The inputs are used to generate a hash which makes up the hash
> representing the task. When the hash changes, we know that sstate is
> invalid and we need to rebuild.
thanks, richard, i appreciate the time you took to clarify this. i'm
going to paraphrase and add it to my WIP "OE/YP best practices"
internal doc; this advice would be contained in the chapter, "for the
love of god, stop doing this!"
thanks again.
rday
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193903):
https://lists.openembedded.org/g/openembedded-core/message/193903
Mute This Topic: https://lists.openembedded.org/mt/103762271/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
