Thanks Martin and Mike for your explanations and tips.
So, I've done a lot of testing today and it seems I simplified the
example in my first email a bit too much. The example as-is works fine
when switching DISTROs as far as I can tell. The problem only arises
when wildcards are used.
Changing my initial example like this should trigger the behaviour I've
initially described:
SRC_URI:append:mydistro-dev = " file://application-dbg.service"
do_install {
# ...snip...
# systemd service
install -d ${D}${systemd_system_unitdir}
install -m 0644 ${WORKDIR}/*.service ${D}${systemd_system_unitdir}
}
do_install:append:mydistro-dev() {
# debug systemd services
install -d ${D}${systemd_system_unitdir}
install -m 0644 ${WORKDIR}/application-dbg.service
${D}${systemd_system_unitdir}
}
Notice the *.service in do_install.
>From my testing, this is how contamination happens:
1) Build with 'DISTRO=mydistro bitbake application'. All tasks for the
recipe are run and the directories in WORKDIR are populated, including
the "application.service" file.
2) Build with 'DISTRO=mydistro-dev bitbake application'. do_unpack is
rerun and places the additional "application-dbg.service" file in
WORKDIR.
3) Switching back to 'mydistro' will get the recipe from sstate cache,
which works fine.
4) Changing application.bb and rebuilding with 'DISTRO=mydistro bitbake
application' reruns do_install (as expected). This leads to the
packages do_install picking up the additional "application-dbg.service"
file left behind by the invocation in step 2).
Mike, Martin: Do you remember in which cases you encountered problems
when sharing the build directory?
On Tue, 2022-07-12 at 09:15 -0700, Khoi Dinh Trinh wrote:
> Thank you Nicolas for asking this question since I will probably run
> into this issue soon if not for this email thread. The answers so far
> have been very helpful but I just want to clarify a bit more on why
> doesn't the package get rebuilt? From my understanding, Yocto should
> rerun a task when the signature of the task changes and since
> do_install has an override on mydistro-dev, shouldn't the content and
> thus the signature of do_install change when switching distro and so
> Yocto should rerun it?
As far as I can tell, all the relevant tasks are rerun correctly when
something is changed. Relevant tasks meaning only the tasks that are
actually different.
The specific issue I experienced is due to the WORKDIR not being
cleaned between different task invocations and the recipe (probably
wrongfully) relying on wildcards to gather files, see example above.
> I have a lot of tasks with override not just on DISTRO but other
> things like MACHINE or custom variables so I want to understand the
> rebuild mechanism as best I can.
>
There's surely someone more knowledgeable here that could clarify the
inner workings of this mechanism a lot better than me.
> Best,
> Khoi Trinh
>
> On Tue, Jul 12, 2022 at 8:05 AM Mike Looijmans
> <mike.looijm...@topic.nl> wrote:
> > Quick answer: Don't build multiple distros in one build directory.
> >
It's really a bummer that it's not reliably possible to switch between
DISTROs inside one build directory.
> > You might get away with setting TMPDIR = "tmp-${DISTRO}" to give
> > each
> > its own.
> >
> > But I'd rather advice to set up two separate builds and just point
> > the
> > downloads and sstate-cache to the same location. It'll be faster
> > than
> > the TMPDIR option.
> >
> > Or figure out how to put the difference in the IMAGE only. Then you
> > can
> > just build both images (in parallel, woot), which is faster, more
> > convenient and saves on diskspace.
As I'm currently working with something close to what you describe, I
think I'll try to stay away from multiple DISTROs if possible and
improve on what I'm already doing.
> >
> > What I often do is have my-application.bb generate a
> > my-application-utils package that only gets installed in the "dev"
> > image
> > but not in the production one, which only installs "my-
> > application".
This is probably what Martin meant with his A.bb and A-xx.bb example.
It's so far one of the best approaches I've seen, thanks.
> >
> > You could also create a "my-application-dev.bb" recipe that
> > includes
> > my-application.bb and just changes what it needs to be different.
> >
> >
> >
> >
> > Met vriendelijke groet / kind regards,
> >
> > Mike Looijmans
> > System Expert
> >
> >
> > TOPIC Embedded Products B.V.
> > Materiaalweg 4, 5681 RJ Best
> > The Netherlands
> >
> > T: +31 (0) 499 33 69 69
> > E: mike.looijm...@topicproducts.com
> > W: www.topic.nl
> >
> > Please consider the environment before printing this e-mail
> > On 12-07-2022 15:37, Nicolas Jeker via lists.yoctoproject.org
> > wrote:
> > > Hi all,
> > >
> > > I'm currently using an additional layer and image to
> > > differentiate
> > > between a release and development build (enabling NFS, SSH, root
> > > login,
> > > etc.). To create a development build, I manually add the layer to
> > > bblayers.conf. This works quite well, but feels a bit clumsy to
> > > integrate into a CI/CD pipeline.
> > >
> > > Per these past discussions here [1][2], I'm now trying to migrate
> > > to
> > > multiple DISTROs, something like "mydistro" and "mydistro-dev".
> > >
> > > While migrating some of the changes, I discovered that I run into
> > > caching(?) issues. I have a recipe for an internal application
> > > and want
> > > to include additional systemd service files in the development
> > > image.
> > >
> > > What I did was this:
> > >
> > > Added "application-dbg.service" to recipes-
> > > internal/application/files
> > >
> > > Adapted application.bb recipe:
> > >
> > > SRC_URI:append:mydistro-dev = " file://application-dbg.service"
> > >
> > > do_install {
> > > # ...snip...
> > > # systemd service
> > > install -d ${D}${systemd_system_unitdir}
> > > install -m 0644 ${WORKDIR}/application.service
> > > ${D}${systemd_system_unitdir}
> > > }
> > >
> > > do_install:append:mydistro-dev() {
> > > # debug systemd services
> > > install -d ${D}${systemd_system_unitdir}
> > > install -m 0644 ${WORKDIR}/application-dbg.service
> > > ${D}${systemd_system_unitdir}
> > > }
> > >
> > >
> > > When I run "DISTRO=mydistro-dev bitbake application" followed by
> > > "DISTRO=mydistro bitbake application", the debug service file is
> > > still
> > > present in the package. This seems to be caused by the "image"
> > > directory in the recipe WORKDIR not being cleaned between
> > > subsequent
> > > do_install runs. Is this expected behaviour? What's the best
> > > solution?
> > >
> > > Kind regards,
> > > Nicolas
> > >
> > > [1]:
> > > https://lore.kernel.org/yocto/CAH9dsRiArf_9GgQS4hCg5=j_jk6cd3eigaoiqd788+isltu...@mail.gmail.com/
> > > [2]:
> > > https://lore.kernel.org/yocto/vi1pr0602mb3549f83ac93a53785de48677d3...@vi1pr0602mb3549.eurprd06.prod.outlook.com/
> > >
> > >
> > >
> >
> > --
> > Mike Looijmans
> >
> >
> >
> >
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#57522): https://lists.yoctoproject.org/g/yocto/message/57522
Mute This Topic: https://lists.yoctoproject.org/mt/92332946/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
