On Fri, Jul 07, 2023 at 10:14:54AM +0100, Jose Quaresma wrote: > Denys Dmytriyenko <de...@denix.org> escreveu no dia quinta, 6/07/2023 à(s) > 19:39: > > > As you noted yourself in that thread, your patches were workarounds and > > just > > masking the actual issue w/o clearly articulating it. The actual race > > issue > > has been under investigation for quite some time and I've discussed it > > with > > Richard before. Luckily he was finally able to root-cause and fix it! > > > > I have reverted my hack to unshare the tmp directory in my distro when > using the oe-core master branch > after the recent spdx changes [1] and everything seems to be okay on my > tests. > The Richard bitbake multiconfig patch to fix the licence collision is for
Couple comments: 1. This was not a mere "license collision", it could affect any deferred task and there were tons of them when building lots of multiconfigs - you'd see a lot of corresponding Notices scrolling by[0]. I've hit it few times in the past and mentioned it to Richard. [0] https://lists.openembedded.org/g/bitbake-devel/topic/99974646 2. We are not using SPDX class here. But from the description of the commit that you've linked I see a common generic issue of overlapping files causing manifest errors. But normally you'd see them when trying to re-use temp dir and/or deploy dir between the builds for different machines... > sure another help to use the shared tmp folder. > However on kisstone I still use the tmp unshared between machines in > multiconfig because the problems were many and > I still don't feel safe to revert to the method used on meta-ti kirkstone. > > [1] > https://git.yoctoproject.org/poky/commit/meta/classes/create-spdx-2.2.bbclass?id=42071227f6f54055d8ac44126ab1d95f83f5b264 > > > > BTW, I'm working on a change that I'm about to submit here, which > > indirectly > > gives you separate TMPDIR for k3r5 - stay tuned. > > > > I'm looking forward to it because the hack I'm using is really ugly and I'd > like to drop it also on the kirkstone branch. > > Cheers, > Jose > > > > > > > > On Wed, Jul 05, 2023 at 09:17:44AM +0100, Jose Quaresma wrote: > > > Just for your information there are more complaints about that issue [1] > > of > > > some bad side effects of sharing the TMP folder in multi configs. > > > > > > [1] https://lists.openembedded.org/g/bitbake-devel/message/14860 > > > > > > Jose > > > > > > Ricardo Salveti <rica...@foundries.io> escreveu no dia segunda, > > 30/01/2023 > > > à(s) 18:27: > > > > > > > On Mon, Jan 30, 2023 at 2:57 PM Jose Quaresma <quaresma.j...@gmail.com > > > > > > > wrote: > > > > > Denys Dmytriyenko <de...@denix.org> escreveu no dia segunda, > > 30/01/2023 > > > > à(s) 17:18: > > > > >> > > > > >> On Mon, Jan 30, 2023 at 11:02:25AM -0600, Ryan Eatmon via > > > > lists.yoctoproject.org wrote: > > > > >> > > > > > >> > > > > > >> > On 1/30/2023 9:20, Jose Quaresma wrote: > > > > >> > >This patch adds the possibility to change some bitbake variable > > > > >> > >that cannot be changed anywhere else, one of then is the TMPDIR. > > > > >> > > > > > > >> > >Using the same TMPDIR it is very sensitive and prone to several > > > > errors > > > > >> > >when used in more complex situations. > > > > >> > >This configuration forces that all native packages have to be the > > > > same between all > > > > >> > >machines and this requirement is very easy to break. > > > > >> > >Suppose you use a macinhe override somewhere on a native recipe > > > > >> > > > > >> Even w/o multiconfig, this will break for multi-machine builds. > > Native > > > > >> packages are for the build host and have no knowledge about the > > target. > > > > >> You should probably look into cross packages instead of native ones > > for > > > > >> this use case. > > > > >> > > > > >> Altering native packages between different targets will cause them > > to > > > > >> rebuild, which they shouldn't. Moreover, it will mess up your sstate > > > > >> and lead to weird issues down the road when trying to re-use your > > > > >> sstate mirror/cache. > > > > > > > > > > Is that the issue of rebuilding some native packages that I am > > trying to > > > > avoid. > > > > > But it is very hard to fix this in OE-core and everything else layers > > > > what happens. > > > > > > > > > > For your information I have some other issues with multiconfig and > > > > native packages > > > > > that have the source files provided on the layer. this two recipes > > don't > > > > work with the > > > > > rm_work bbclass because the in one of the machines > > > > > texinfo-dummy-native gettext-minimal-native > > > > > > > > > > Another issue is that sometimes one of the machines uses the > > > > SOURCE_DATE_EPOCH_FALLBACK > > > > > and the other uses SOURCE_DATE_EPOCH and this also triggers a > > rebuild > > > > of one of the machines. > > > > > > > > > > Recently a new issue with the rm_work and the spdx > > > > > comes in > > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > where I have the steps to reproduce. > > > > > > > > > > All of these issues can be solved easily with TMDIR for each machine, > > > > > but because I understand it can be invasive I submit this patch that > > > > doesn't change anything > > > > > in meta-ti but can add the possibility to use a different TMPDIR for > > > > each machine. > > > > > > > > > > A different TMPDIR for each machine is also referenced in the > > official > > > > documentation > > > > > "Minimally, each configuration file must define the machine and the > > > > temporary directory BitBake uses for the build. Suggested practice > > dictates > > > > that you do not overlap the temporary directories used during the > > builds." > > > > > > > > > > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build > > > > > > > > > >> > >and this requirement is no longer met. > > > > >> > >Many of these anomalies can be verified with the use of the > > rm_work > > > > and > > > > >> > >create-spdx bitbake classes. > > > > >> > > > > > > >> > >A previous attempt [1] had already been made but was refused > > > > >> > >but this way it can be done without side effects for other users. > > > > >> > > > > > > >> > >[1] https://lists.yoctoproject.org/g/meta-ti/message/14767 > > > > >> > > > > > >> > > > > > >> > So the difference between this patch and the previous one, is that > > > > >> > you are now included a file that is not there normally so that you > > > > >> > can put the contents of the previous patch in the new include file > > > > >> > to get around the issue in the archiver. > > > > > > > > > > The archiver issue is fixed but it will be very easy to break again. > > > > > Myself recently broke it again so I added a test to help there > > > > > > > > > > > https://git.yoctoproject.org/poky/commit/?id=b5baa7dc8bd1c0d2d78c532d97a44120346b5edd > > > > > > > > > >> > > > > > >> > I'm not saying I'm opposed to this approach (yet), I'm just trying > > > > >> > to understand the entire story. > > > > >> > > > > > >> > Did you find anything in looking at fixing the archiver to better > > > > >> > support the multi-config issue? > > > > > > > > > > The issue now is different and is related with spdx and rm_work > > > > > https://lists.openembedded.org/g/openembedded-core/message/174276 > > > > > > > > > > With this patch I can set a new TMPDIR folder for each machine and > > > > > can build successfully. > > > > > > > > If you don't have spdx enabled then these sort of issues happen less > > > > often, but after the above patch was merged in oe-core (and also > > > > backported to kirkstone), we cannot build multiconfig targets from > > > > meta-ti anymore with spdx enabled. > > > > > > > > This is special to spdx because of the way the task dependencies are > > > > done, which requires it to be executed before rm_work. This is OK when > > > > a machine specific TMPDIR is used and explodes quite quickly (due > > > > races) when sharing TMPDIR because of the parallel rm_work tasks that > > > > end up removing stuff that is still required by other parallel tasks.
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16834): https://lists.yoctoproject.org/g/meta-ti/message/16834 Mute This Topic: https://lists.yoctoproject.org/mt/96629864/21656 Group Owner: meta-ti+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/meta-ti/leave/6695321/21656/1393940836/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
