Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
Hi,
v4 fixes the issue
Jose Quaresma via lists.openembedded.org escreveu no dia terça, 14/06/2022 à(s)
10:43:
>
>
> Jose Quaresma via lists.openembedded.org gmail@lists.openembedded.org> escreveu no dia segunda, 13/06/2022
> à(s) 23:12:
>
>> Hi Richard,
>>
>> Richard Purdie escreveu no dia
>> sexta, 10/06/2022 à(s) 11:59:
>>
>>> On Fri, 2022-06-10 at 11:43 +0100, Richard Purdie via
>>> lists.openembedded.org wrote:
>>> > On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
>>> > > The archiver doesn't support using multiconfig sharing the same
>>> TMPDIR.
>>> > > When in the multiconfig and using the same TMPDIR: the two machines
>>> can make
>>> > > the same copy at the same time to the same destination, which will
>>> not work.
>>> > >
>>> > > This issiue can be replicated with meta-ti layer:
>>> > > > # use a multiconfig machine
>>> > > > MACHINE = "am64xx-evm"
>>> > > > # archive sources for target recipes in conf/local.conf
>>> > > > INHERIT += "archiver"
>>> > > > COPYLEFT_RECIPE_TYPES = "target"
>>> > > > ARCHIVER_MODE[src] = "original"
>>> > > > ARCHIVER_MODE[diff] = "1"
>>> > > > # and run bitbake with:
>>> > > > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c
>>> do_unpack_and_patch -f
>>> > >
>>> >
>>> > I went digging. If I enable this as above (using master) and run
>>> >
>>> > $ bitbake mc:k3r5:gcc-source-12.1.0 gcc-source-12.1.0 -c
>>> do_unpack_and_patch -S none
>>> >
>>> > then:
>>> >
>>> > $ bitbake-diffsigs
>>> tmp/stamps/work-shared/gcc-12.1.0-r0.do_unpack_and_patch.sigdata.*
>>> > NOTE: Starting bitbake server...
>>> > Task dependencies changed from:
>>> > ['ABIEXTENSION', 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]',
>>> 'ARCHIVER_MODE[src]', 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR',
>>> 'ARCHIVER_WORKDIR', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
>>> 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
>>> 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
>>> 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
>>> 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
>>> 'TARGET_VENDOR', 'TUNE_ARCH', 'UNPACK_DIR', 'base_do_unpack',
>>> 'base_do_unpack[cleandirs]', 'create_diff_gz',
>>> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
>>> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
>>> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
>>> > to:
>>> > ['ABIEXTENSION', 'ABIEXTENSION_32', 'ABIEXTENSION_64',
>>> 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]', 'ARCHIVER_MODE[src]',
>>> 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
>>> 'ARMPKGSFX_ENDIAN_64', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR',
>>> 'HOST_ARCH', 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP',
>>> 'LIBCEXTENSION', 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV',
>>> 'RECIPE_SYSROOT_NATIVE', 'S', 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR',
>>> 'SRC_URI', 'SRC_URI[sha256sum]', 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH',
>>> 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TUNE_ARCH', 'TUNE_ARCH_32',
>>> 'TUNE_ARCH_64', 'UNPACK_DIR', 'base_do_unpack',
>>> 'base_do_unpack[cleandirs]', 'create_diff_gz',
>>> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
>>> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
>>> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
>>> > basehash changed from
>>> 005580a7a5a1ab1ff554d0a9e47c2ad4ba932f3fe185d8d349b06c83e5e58b6c to
>>> d7b2e696005a578d5958ef4e73f1fc83e22612a3ae4294fb0540b0fadcb6c50f
>>> > List of dependencies for variable ABIEXTENSION changed from 'set()' to
>>> '{'ABIEXTENSION_64', 'ABIEXTENSION_32'}'
>>> > List of dependencies for variable TUNE_ARCH changed from 'set()' to
>>> '{'TUNE_ARCH_32', 'TUNE_ARCH_64'}'
>>> > Dependency on variable ABIEXTENSION_32 was added
>>> > Dependency on variable ABIEXTENSION_64 was added
>>> > Dependency on variable ARMPKGSFX_ENDIAN_64 was added
>>> > Dependency on variable TUNE_ARCH_32 was added
>>> > Dependency on variable TUNE_ARCH_64 was added
>>> > Variable ABIEXTENSION value changed:
>>> > @@ -1 +1,2 @@
>>> > -eabi
>>> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64',
>>> '${ABIEXTENSION_64}', '${ABIEXTENSION_32}', d)}
>>> > +TUNE_FEATURES{aarch64} = Set
>>> > Variable TUNE_ARCH value changed:
>>> > @@ -1,2 +1,2 @@
>>> > -${@bb.utils.contains('TUNE_FEATURES', 'bigendian', 'armeb', 'arm', d)}
>>> > -TUNE_FEATURES{bigendian} = Unset
>>> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${TUNE_ARCH_64}',
>>> '${TUNE_ARCH_32}', d)}
>>> > +TUNE_FEATURES{aarch64} = Set
>>> >
>>> > this tells us the real issue. The task is changing due to toolchain
>>> > issues. These stamps should be the same and aren't and this is the real
>>> > issue that needs fixing.
>>>
>>
>> Many thanks for this tip that helps me a lot.
>> I will use this procedure to see what are the changes in some other
>> issues in the rm_work class that I found as well related with multi
Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
Jose Quaresma via lists.openembedded.org escreveu no dia segunda, 13/06/2022 à(s)
23:12:
> Hi Richard,
>
> Richard Purdie escreveu no dia
> sexta, 10/06/2022 à(s) 11:59:
>
>> On Fri, 2022-06-10 at 11:43 +0100, Richard Purdie via
>> lists.openembedded.org wrote:
>> > On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
>> > > The archiver doesn't support using multiconfig sharing the same
>> TMPDIR.
>> > > When in the multiconfig and using the same TMPDIR: the two machines
>> can make
>> > > the same copy at the same time to the same destination, which will
>> not work.
>> > >
>> > > This issiue can be replicated with meta-ti layer:
>> > > > # use a multiconfig machine
>> > > > MACHINE = "am64xx-evm"
>> > > > # archive sources for target recipes in conf/local.conf
>> > > > INHERIT += "archiver"
>> > > > COPYLEFT_RECIPE_TYPES = "target"
>> > > > ARCHIVER_MODE[src] = "original"
>> > > > ARCHIVER_MODE[diff] = "1"
>> > > > # and run bitbake with:
>> > > > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c
>> do_unpack_and_patch -f
>> > >
>> >
>> > I went digging. If I enable this as above (using master) and run
>> >
>> > $ bitbake mc:k3r5:gcc-source-12.1.0 gcc-source-12.1.0 -c
>> do_unpack_and_patch -S none
>> >
>> > then:
>> >
>> > $ bitbake-diffsigs
>> tmp/stamps/work-shared/gcc-12.1.0-r0.do_unpack_and_patch.sigdata.*
>> > NOTE: Starting bitbake server...
>> > Task dependencies changed from:
>> > ['ABIEXTENSION', 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]',
>> 'ARCHIVER_MODE[src]', 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR',
>> 'ARCHIVER_WORKDIR', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
>> 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
>> 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
>> 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
>> 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
>> 'TARGET_VENDOR', 'TUNE_ARCH', 'UNPACK_DIR', 'base_do_unpack',
>> 'base_do_unpack[cleandirs]', 'create_diff_gz',
>> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
>> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
>> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
>> > to:
>> > ['ABIEXTENSION', 'ABIEXTENSION_32', 'ABIEXTENSION_64',
>> 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]', 'ARCHIVER_MODE[src]',
>> 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
>> 'ARMPKGSFX_ENDIAN_64', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR',
>> 'HOST_ARCH', 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP',
>> 'LIBCEXTENSION', 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV',
>> 'RECIPE_SYSROOT_NATIVE', 'S', 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR',
>> 'SRC_URI', 'SRC_URI[sha256sum]', 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH',
>> 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TUNE_ARCH', 'TUNE_ARCH_32',
>> 'TUNE_ARCH_64', 'UNPACK_DIR', 'base_do_unpack',
>> 'base_do_unpack[cleandirs]', 'create_diff_gz',
>> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
>> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
>> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
>> > basehash changed from
>> 005580a7a5a1ab1ff554d0a9e47c2ad4ba932f3fe185d8d349b06c83e5e58b6c to
>> d7b2e696005a578d5958ef4e73f1fc83e22612a3ae4294fb0540b0fadcb6c50f
>> > List of dependencies for variable ABIEXTENSION changed from 'set()' to
>> '{'ABIEXTENSION_64', 'ABIEXTENSION_32'}'
>> > List of dependencies for variable TUNE_ARCH changed from 'set()' to
>> '{'TUNE_ARCH_32', 'TUNE_ARCH_64'}'
>> > Dependency on variable ABIEXTENSION_32 was added
>> > Dependency on variable ABIEXTENSION_64 was added
>> > Dependency on variable ARMPKGSFX_ENDIAN_64 was added
>> > Dependency on variable TUNE_ARCH_32 was added
>> > Dependency on variable TUNE_ARCH_64 was added
>> > Variable ABIEXTENSION value changed:
>> > @@ -1 +1,2 @@
>> > -eabi
>> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${ABIEXTENSION_64}',
>> '${ABIEXTENSION_32}', d)}
>> > +TUNE_FEATURES{aarch64} = Set
>> > Variable TUNE_ARCH value changed:
>> > @@ -1,2 +1,2 @@
>> > -${@bb.utils.contains('TUNE_FEATURES', 'bigendian', 'armeb', 'arm', d)}
>> > -TUNE_FEATURES{bigendian} = Unset
>> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${TUNE_ARCH_64}',
>> '${TUNE_ARCH_32}', d)}
>> > +TUNE_FEATURES{aarch64} = Set
>> >
>> > this tells us the real issue. The task is changing due to toolchain
>> > issues. These stamps should be the same and aren't and this is the real
>> > issue that needs fixing.
>>
>
> Many thanks for this tip that helps me a lot.
> I will use this procedure to see what are the changes in some other issues
> in the rm_work class that I found as well related with multiconfig.
> But first is the gcc-sources shared recipe.
>
>
>>
>> To further follow up, there are two issues:
>>
>> a) the task uses ${B} which for gcc includes HOST_SYS and TARGET_SYS.
>> We can just set that in the gcc-source recipe to B =
>> "${WORKDIR}/bu
Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
Hi Richard,
Richard Purdie escreveu no dia sexta,
10/06/2022 à(s) 11:59:
> On Fri, 2022-06-10 at 11:43 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
> > > The archiver doesn't support using multiconfig sharing the same TMPDIR.
> > > When in the multiconfig and using the same TMPDIR: the two machines
> can make
> > > the same copy at the same time to the same destination, which will not
> work.
> > >
> > > This issiue can be replicated with meta-ti layer:
> > > > # use a multiconfig machine
> > > > MACHINE = "am64xx-evm"
> > > > # archive sources for target recipes in conf/local.conf
> > > > INHERIT += "archiver"
> > > > COPYLEFT_RECIPE_TYPES = "target"
> > > > ARCHIVER_MODE[src] = "original"
> > > > ARCHIVER_MODE[diff] = "1"
> > > > # and run bitbake with:
> > > > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c
> do_unpack_and_patch -f
> > >
> >
> > I went digging. If I enable this as above (using master) and run
> >
> > $ bitbake mc:k3r5:gcc-source-12.1.0 gcc-source-12.1.0 -c
> do_unpack_and_patch -S none
> >
> > then:
> >
> > $ bitbake-diffsigs
> tmp/stamps/work-shared/gcc-12.1.0-r0.do_unpack_and_patch.sigdata.*
> > NOTE: Starting bitbake server...
> > Task dependencies changed from:
> > ['ABIEXTENSION', 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]',
> 'ARCHIVER_MODE[src]', 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR',
> 'ARCHIVER_WORKDIR', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
> 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
> 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
> 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
> 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
> 'TARGET_VENDOR', 'TUNE_ARCH', 'UNPACK_DIR', 'base_do_unpack',
> 'base_do_unpack[cleandirs]', 'create_diff_gz',
> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
> > to:
> > ['ABIEXTENSION', 'ABIEXTENSION_32', 'ABIEXTENSION_64',
> 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]', 'ARCHIVER_MODE[src]',
> 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
> 'ARMPKGSFX_ENDIAN_64', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR',
> 'HOST_ARCH', 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP',
> 'LIBCEXTENSION', 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV',
> 'RECIPE_SYSROOT_NATIVE', 'S', 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR',
> 'SRC_URI', 'SRC_URI[sha256sum]', 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH',
> 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TUNE_ARCH', 'TUNE_ARCH_32',
> 'TUNE_ARCH_64', 'UNPACK_DIR', 'base_do_unpack',
> 'base_do_unpack[cleandirs]', 'create_diff_gz',
> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch',
> 'do_qa_unpack', 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared',
> 'patch_do_patch', 'src_patches', 'unpack_check_src_uri']
> > basehash changed from
> 005580a7a5a1ab1ff554d0a9e47c2ad4ba932f3fe185d8d349b06c83e5e58b6c to
> d7b2e696005a578d5958ef4e73f1fc83e22612a3ae4294fb0540b0fadcb6c50f
> > List of dependencies for variable ABIEXTENSION changed from 'set()' to
> '{'ABIEXTENSION_64', 'ABIEXTENSION_32'}'
> > List of dependencies for variable TUNE_ARCH changed from 'set()' to
> '{'TUNE_ARCH_32', 'TUNE_ARCH_64'}'
> > Dependency on variable ABIEXTENSION_32 was added
> > Dependency on variable ABIEXTENSION_64 was added
> > Dependency on variable ARMPKGSFX_ENDIAN_64 was added
> > Dependency on variable TUNE_ARCH_32 was added
> > Dependency on variable TUNE_ARCH_64 was added
> > Variable ABIEXTENSION value changed:
> > @@ -1 +1,2 @@
> > -eabi
> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${ABIEXTENSION_64}',
> '${ABIEXTENSION_32}', d)}
> > +TUNE_FEATURES{aarch64} = Set
> > Variable TUNE_ARCH value changed:
> > @@ -1,2 +1,2 @@
> > -${@bb.utils.contains('TUNE_FEATURES', 'bigendian', 'armeb', 'arm', d)}
> > -TUNE_FEATURES{bigendian} = Unset
> > +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${TUNE_ARCH_64}',
> '${TUNE_ARCH_32}', d)}
> > +TUNE_FEATURES{aarch64} = Set
> >
> > this tells us the real issue. The task is changing due to toolchain
> > issues. These stamps should be the same and aren't and this is the real
> > issue that needs fixing.
>
Many thanks for this tip that helps me a lot.
I will use this procedure to see what are the changes in some other issues
in the rm_work class that I found as well related with multiconfig.
But first is the gcc-sources shared recipe.
>
> To further follow up, there are two issues:
>
> a) the task uses ${B} which for gcc includes HOST_SYS and TARGET_SYS.
> We can just set that in the gcc-source recipe to B =
> "${WORKDIR}/build".
>
> b) the archiver uses:
> ARCHIVER_OUTDIR = ${ARCHIVER_TOPDIR}/${TARGET_SYS}/${PF}/
> which includes TARGET_SYS and between the two different MACHINE values,
> this changes from 'arm-poky-linux-gnueab
Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
On Fri, 2022-06-10 at 11:43 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
> > The archiver doesn't support using multiconfig sharing the same TMPDIR.
> > When in the multiconfig and using the same TMPDIR: the two machines can make
> > the same copy at the same time to the same destination, which will not work.
> >
> > This issiue can be replicated with meta-ti layer:
> > > # use a multiconfig machine
> > > MACHINE = "am64xx-evm"
> > > # archive sources for target recipes in conf/local.conf
> > > INHERIT += "archiver"
> > > COPYLEFT_RECIPE_TYPES = "target"
> > > ARCHIVER_MODE[src] = "original"
> > > ARCHIVER_MODE[diff] = "1"
> > > # and run bitbake with:
> > > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c
> > > do_unpack_and_patch -f
> >
>
> I went digging. If I enable this as above (using master) and run
>
> $ bitbake mc:k3r5:gcc-source-12.1.0 gcc-source-12.1.0 -c do_unpack_and_patch
> -S none
>
> then:
>
> $ bitbake-diffsigs
> tmp/stamps/work-shared/gcc-12.1.0-r0.do_unpack_and_patch.sigdata.*
> NOTE: Starting bitbake server...
> Task dependencies changed from:
> ['ABIEXTENSION', 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]',
> 'ARCHIVER_MODE[src]', 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR',
> 'ARCHIVER_WORKDIR', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
> 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
> 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
> 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
> 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
> 'TARGET_VENDOR', 'TUNE_ARCH', 'UNPACK_DIR', 'base_do_unpack',
> 'base_do_unpack[cleandirs]', 'create_diff_gz',
> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch', 'do_qa_unpack',
> 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared', 'patch_do_patch',
> 'src_patches', 'unpack_check_src_uri']
> to:
> ['ABIEXTENSION', 'ABIEXTENSION_32', 'ABIEXTENSION_64',
> 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]', 'ARCHIVER_MODE[src]',
> 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
> 'ARMPKGSFX_ENDIAN_64', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
> 'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
> 'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
> 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
> 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
> 'TARGET_VENDOR', 'TUNE_ARCH', 'TUNE_ARCH_32', 'TUNE_ARCH_64', 'UNPACK_DIR',
> 'base_do_unpack', 'base_do_unpack[cleandirs]', 'create_diff_gz',
> 'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch', 'do_qa_unpack',
> 'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared', 'patch_do_patch',
> 'src_patches', 'unpack_check_src_uri']
> basehash changed from
> 005580a7a5a1ab1ff554d0a9e47c2ad4ba932f3fe185d8d349b06c83e5e58b6c to
> d7b2e696005a578d5958ef4e73f1fc83e22612a3ae4294fb0540b0fadcb6c50f
> List of dependencies for variable ABIEXTENSION changed from 'set()' to
> '{'ABIEXTENSION_64', 'ABIEXTENSION_32'}'
> List of dependencies for variable TUNE_ARCH changed from 'set()' to
> '{'TUNE_ARCH_32', 'TUNE_ARCH_64'}'
> Dependency on variable ABIEXTENSION_32 was added
> Dependency on variable ABIEXTENSION_64 was added
> Dependency on variable ARMPKGSFX_ENDIAN_64 was added
> Dependency on variable TUNE_ARCH_32 was added
> Dependency on variable TUNE_ARCH_64 was added
> Variable ABIEXTENSION value changed:
> @@ -1 +1,2 @@
> -eabi
> +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${ABIEXTENSION_64}',
> '${ABIEXTENSION_32}', d)}
> +TUNE_FEATURES{aarch64} = Set
> Variable TUNE_ARCH value changed:
> @@ -1,2 +1,2 @@
> -${@bb.utils.contains('TUNE_FEATURES', 'bigendian', 'armeb', 'arm', d)}
> -TUNE_FEATURES{bigendian} = Unset
> +${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${TUNE_ARCH_64}',
> '${TUNE_ARCH_32}', d)}
> +TUNE_FEATURES{aarch64} = Set
>
> this tells us the real issue. The task is changing due to toolchain
> issues. These stamps should be the same and aren't and this is the real
> issue that needs fixing.
To further follow up, there are two issues:
a) the task uses ${B} which for gcc includes HOST_SYS and TARGET_SYS.
We can just set that in the gcc-source recipe to B =
"${WORKDIR}/build".
b) the archiver uses:
ARCHIVER_OUTDIR = ${ARCHIVER_TOPDIR}/${TARGET_SYS}/${PF}/
which includes TARGET_SYS and between the two different MACHINE values,
this changes from 'arm-poky-linux-gnueabi' to 'aarch64-poky-linux'.
This leads to the task running twice, once for each multiconfig.
The first is easily solved as I mentioned. The question in the second
case is whether the task should run twice and where the output should
go since it is common to both builds. I do wonder if the archiver
shouldn't set:
ARCHIVER_OUTPUT:pn-gcc-source = "${ARCHIVER_TOPDIR}/allarch/${PF}"
Cheers,
Richard
Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
> The archiver doesn't support using multiconfig sharing the same TMPDIR.
> When in the multiconfig and using the same TMPDIR: the two machines can make
> the same copy at the same time to the same destination, which will not work.
>
> This issiue can be replicated with meta-ti layer:
> > # use a multiconfig machine
> > MACHINE = "am64xx-evm"
> > # archive sources for target recipes in conf/local.conf
> > INHERIT += "archiver"
> > COPYLEFT_RECIPE_TYPES = "target"
> > ARCHIVER_MODE[src] = "original"
> > ARCHIVER_MODE[diff] = "1"
> > # and run bitbake with:
> > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch
> > -f
>
I went digging. If I enable this as above (using master) and run
$ bitbake mc:k3r5:gcc-source-12.1.0 gcc-source-12.1.0 -c do_unpack_and_patch -S
none
then:
$ bitbake-diffsigs
tmp/stamps/work-shared/gcc-12.1.0-r0.do_unpack_and_patch.sigdata.*
NOTE: Starting bitbake server...
Task dependencies changed from:
['ABIEXTENSION', 'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]',
'ARCHIVER_MODE[src]', 'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH', 'HOST_OS', 'HOST_SYS',
'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION', 'PATCHTOOL', 'PE', 'PF', 'PN',
'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S', 'SDE_DIR', 'SDE_FILE', 'SOURCEDIR',
'SRC_URI', 'SRC_URI[sha256sum]', 'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH',
'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TUNE_ARCH', 'UNPACK_DIR',
'base_do_unpack', 'base_do_unpack[cleandirs]', 'create_diff_gz',
'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch', 'do_qa_unpack',
'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared', 'patch_do_patch',
'src_patches', 'unpack_check_src_uri']
to:
['ABIEXTENSION', 'ABIEXTENSION_32', 'ABIEXTENSION_64',
'ARCHIVER_MODE[diff-exclude]', 'ARCHIVER_MODE[diff]', 'ARCHIVER_MODE[src]',
'ARCHIVER_OUTDIR', 'ARCHIVER_TOPDIR', 'ARCHIVER_WORKDIR',
'ARMPKGSFX_ENDIAN_64', 'B', 'BASEURI', 'EXTENDPE', 'GNU_MIRROR', 'HOST_ARCH',
'HOST_OS', 'HOST_SYS', 'HOST_VENDOR', 'INSANE_SKIP', 'LIBCEXTENSION',
'PATCHTOOL', 'PE', 'PF', 'PN', 'PR', 'PV', 'RECIPE_SYSROOT_NATIVE', 'S',
'SDE_DIR', 'SDE_FILE', 'SOURCEDIR', 'SRC_URI', 'SRC_URI[sha256sum]',
'STAGING_DIR_NATIVE', 'T', 'TARGET_ARCH', 'TARGET_OS', 'TARGET_SYS',
'TARGET_VENDOR', 'TUNE_ARCH', 'TUNE_ARCH_32', 'TUNE_ARCH_64', 'UNPACK_DIR',
'base_do_unpack', 'base_do_unpack[cleandirs]', 'create_diff_gz',
'create_source_date_epoch_stamp', 'do_patch', 'do_qa_patch', 'do_qa_unpack',
'do_unpack', 'do_unpack[cleandirs]', 'is_work_shared', 'patch_do_patch',
'src_patches', 'unpack_check_src_uri']
basehash changed from
005580a7a5a1ab1ff554d0a9e47c2ad4ba932f3fe185d8d349b06c83e5e58b6c to
d7b2e696005a578d5958ef4e73f1fc83e22612a3ae4294fb0540b0fadcb6c50f
List of dependencies for variable ABIEXTENSION changed from 'set()' to
'{'ABIEXTENSION_64', 'ABIEXTENSION_32'}'
List of dependencies for variable TUNE_ARCH changed from 'set()' to
'{'TUNE_ARCH_32', 'TUNE_ARCH_64'}'
Dependency on variable ABIEXTENSION_32 was added
Dependency on variable ABIEXTENSION_64 was added
Dependency on variable ARMPKGSFX_ENDIAN_64 was added
Dependency on variable TUNE_ARCH_32 was added
Dependency on variable TUNE_ARCH_64 was added
Variable ABIEXTENSION value changed:
@@ -1 +1,2 @@
-eabi
+${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${ABIEXTENSION_64}',
'${ABIEXTENSION_32}', d)}
+TUNE_FEATURES{aarch64} = Set
Variable TUNE_ARCH value changed:
@@ -1,2 +1,2 @@
-${@bb.utils.contains('TUNE_FEATURES', 'bigendian', 'armeb', 'arm', d)}
-TUNE_FEATURES{bigendian} = Unset
+${@bb.utils.contains('TUNE_FEATURES', 'aarch64', '${TUNE_ARCH_64}',
'${TUNE_ARCH_32}', d)}
+TUNE_FEATURES{aarch64} = Set
this tells us the real issue. The task is changing due to toolchain
issues. These stamps should be the same and aren't and this is the real
issue that needs fixing.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166795):
https://lists.openembedded.org/g/openembedded-core/message/166795
Mute This Topic: https://lists.openembedded.org/mt/91650171/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
On Thu, 2022-06-09 at 17:39 +0100, Jose Quaresma wrote:
> The archiver doesn't support using multiconfig sharing the same TMPDIR.
> When in the multiconfig and using the same TMPDIR: the two machines can make
> the same copy at the same time to the same destination, which will not work.
>
> This issiue can be replicated with meta-ti layer:
> > # use a multiconfig machine
> > MACHINE = "am64xx-evm"
> > # archive sources for target recipes in conf/local.conf
> > INHERIT += "archiver"
> > COPYLEFT_RECIPE_TYPES = "target"
> > ARCHIVER_MODE[src] = "original"
> > ARCHIVER_MODE[diff] = "1"
> > # and run bitbake with:
> > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch
> > -f
>
> Fixes the following:
> > ERROR: mc:k3r5:gcc-source-11.2.0-11.2.0-r0 do_unpack_and_patch: Error
> > executing a python function in exec_func_python() autogenerated:
> >
> > The stack trace of python calls that resulted in this exception/failure was:
> > File: 'exec_func_python() autogenerated', lineno: 2, function:
> > 0001:
> > *** 0002:do_unpack_and_patch(d)
> > 0003:
> > File:
> > '/build/conf/../../layers/openembedded-core/meta/classes/archiver.bbclass',
> > lineno: 489, function: do_unpack_and_patch
> > 0485:# Save the original source for creating the patches
> > 0486:if d.getVarFlag('ARCHIVER_MODE', 'diff') == '1':
> > 0487:src = d.getVar('S').rstrip('/')
> > 0488:src_orig = '%s.orig' % src
> > *** 0489:oe.path.copytree(src, src_orig)
> > 0490:
> > 0491:if bb.data.inherits_class('dos2unix', d):
> > 0492:bb.build.exec_func('do_convert_crlf_to_lf', d)
> > 0493:
> > File: '/build/conf/../../layers/openembedded-core/meta/lib/oe/path.py',
> > lineno: 94, function: copytree
> > 0090:# This way we also preserve hardlinks between files in the
> > tree.
> > 0091:
> > 0092:bb.utils.mkdirhier(dst)
> > 0093:cmd = "tar --xattrs --xattrs-include='*' -cf - -S -C %s -p . |
> > tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
> > *** 0094:subprocess.check_output(cmd, shell=True,
> > stderr=subprocess.STDOUT)
> > 0095:
> > 0096:def copyhardlinktree(src, dst):
> > 0097:"""Make a tree of hard links when possible, otherwise copy."""
> > 0098:bb.utils.mkdirhier(dst)
> > File: '/usr/lib/python3.10/subprocess.py', lineno: 420, function:
> > check_output
> > 0416:else:
> > 0417:empty = b''
> > 0418:kwargs['input'] = empty
> > 0419:
> > *** 0420:return run(*popenargs, stdout=PIPE, timeout=timeout,
> > check=True,
> > 0421: **kwargs).stdout
> > 0422:
> > 0423:
> > 0424:class CompletedProcess(object):
> > File: '/usr/lib/python3.10/subprocess.py', lineno: 524, function: run
> > 0520:# We don't call process.wait() as .__exit__ does that
> > for us.
> > 0521:raise
> > 0522:retcode = process.poll()
> > 0523:if check and retcode:
> > *** 0524:raise CalledProcessError(retcode, process.args,
> > 0525: output=stdout, stderr=stderr)
> > 0526:return CompletedProcess(process.args, retcode, stdout, stderr)
> > 0527:
> > 0528:
> > Exception: subprocess.CalledProcessError: Command 'tar --xattrs
> > --xattrs-include='*' -cf - -S -C
> > /build/tmp-lmp/work-shared/gcc-11.2.0-r0/gcc-11.2.0 -p . | tar --xattrs
> > --xattrs-include='*' |-xf - -C
> > /build/tmp-lmp/work-shared/gcc-11.2.0-r0/gcc-11.2.0.orig' returned non-zero
> > exit status 2.
> >
> > Subprocess output:
> > tar: ./gcc/config/moxie/predicates.md: file changed as we read it
> > tar: ./patches: Cannot stat: No such file or directory
> > tar: Exiting with failure status due to previous errors
> >
> > ERROR: Logfile of failure stored in:
> > /build/tmp-lmp/work-shared/gcc-11.2.0-r0/temp/log.do_unpack_and_patch.291906
> > ERROR: Task
> > (mc:k3r5:/build/conf/../../layers/openembedded-core/meta/recipes-devtools/gcc/gcc-source_11.2.bb:do_unpack_and_patch)
> > failed with exit code '1'
>
> Signed-off-by: Jose Quaresma
> ---
>
> v2: replace "bb.error" with "bb.fatal"
> v3: instead of failing make it more resilient
>
> meta/classes/archiver.bbclass | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
> index c19c770d11..b1d7aa0247 100644
> --- a/meta/classes/archiver.bbclass
> +++ b/meta/classes/archiver.bbclass
> @@ -485,7 +485,8 @@ python do_unpack_and_patch() {
> # Save the original source for creating the patches
> if d.getVarFlag('ARCHIVER_MODE', 'diff') == '1':
> src = d.getVar('S').rstrip('/')
> -src_orig = '%s.orig' % src
> +# append machine name to avoid races in multiconfig when using the
> same TMPDIR
> +src_orig = '%s.orig.%s' % (src, d.getVar('MACHINE'))
> oe.path.copytree
[OE-core] [PATCH v3] archiver: avoid race conditions when using multiconfig and sharing the same TMPDIR
The archiver doesn't support using multiconfig sharing the same TMPDIR.
When in the multiconfig and using the same TMPDIR: the two machines can make
the same copy at the same time to the same destination, which will not work.
This issiue can be replicated with meta-ti layer:
|# use a multiconfig machine
|MACHINE = "am64xx-evm"
|# archive sources for target recipes in conf/local.conf
|INHERIT += "archiver"
|COPYLEFT_RECIPE_TYPES = "target"
|ARCHIVER_MODE[src] = "original"
|ARCHIVER_MODE[diff] = "1"
|# and run bitbake with:
|bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch -f
Fixes the following:
|ERROR: mc:k3r5:gcc-source-11.2.0-11.2.0-r0 do_unpack_and_patch: Error
executing a python function in exec_func_python() autogenerated:
|
|The stack trace of python calls that resulted in this exception/failure was:
|File: 'exec_func_python() autogenerated', lineno: 2, function:
| 0001:
| *** 0002:do_unpack_and_patch(d)
| 0003:
|File:
'/build/conf/../../layers/openembedded-core/meta/classes/archiver.bbclass',
lineno: 489, function: do_unpack_and_patch
| 0485:# Save the original source for creating the patches
| 0486:if d.getVarFlag('ARCHIVER_MODE', 'diff') == '1':
| 0487:src = d.getVar('S').rstrip('/')
| 0488:src_orig = '%s.orig' % src
| *** 0489:oe.path.copytree(src, src_orig)
| 0490:
| 0491:if bb.data.inherits_class('dos2unix', d):
| 0492:bb.build.exec_func('do_convert_crlf_to_lf', d)
| 0493:
|File: '/build/conf/../../layers/openembedded-core/meta/lib/oe/path.py',
lineno: 94, function: copytree
| 0090:# This way we also preserve hardlinks between files in the tree.
| 0091:
| 0092:bb.utils.mkdirhier(dst)
| 0093:cmd = "tar --xattrs --xattrs-include='*' -cf - -S -C %s -p . |
tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
| *** 0094:subprocess.check_output(cmd, shell=True,
stderr=subprocess.STDOUT)
| 0095:
| 0096:def copyhardlinktree(src, dst):
| 0097:"""Make a tree of hard links when possible, otherwise copy."""
| 0098:bb.utils.mkdirhier(dst)
|File: '/usr/lib/python3.10/subprocess.py', lineno: 420, function: check_output
| 0416:else:
| 0417:empty = b''
| 0418:kwargs['input'] = empty
| 0419:
| *** 0420:return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
| 0421: **kwargs).stdout
| 0422:
| 0423:
| 0424:class CompletedProcess(object):
|File: '/usr/lib/python3.10/subprocess.py', lineno: 524, function: run
| 0520:# We don't call process.wait() as .__exit__ does that
for us.
| 0521:raise
| 0522:retcode = process.poll()
| 0523:if check and retcode:
| *** 0524:raise CalledProcessError(retcode, process.args,
| 0525: output=stdout, stderr=stderr)
| 0526:return CompletedProcess(process.args, retcode, stdout, stderr)
| 0527:
| 0528:
|Exception: subprocess.CalledProcessError: Command 'tar --xattrs
--xattrs-include='*' -cf - -S -C
/build/tmp-lmp/work-shared/gcc-11.2.0-r0/gcc-11.2.0 -p . | tar --xattrs
--xattrs-include='*' |-xf - -C
/build/tmp-lmp/work-shared/gcc-11.2.0-r0/gcc-11.2.0.orig' returned non-zero
exit status 2.
|
|Subprocess output:
|tar: ./gcc/config/moxie/predicates.md: file changed as we read it
|tar: ./patches: Cannot stat: No such file or directory
|tar: Exiting with failure status due to previous errors
|
|ERROR: Logfile of failure stored in:
/build/tmp-lmp/work-shared/gcc-11.2.0-r0/temp/log.do_unpack_and_patch.291906
|ERROR: Task
(mc:k3r5:/build/conf/../../layers/openembedded-core/meta/recipes-devtools/gcc/gcc-source_11.2.bb:do_unpack_and_patch)
failed with exit code '1'
Signed-off-by: Jose Quaresma
---
v2: replace "bb.error" with "bb.fatal"
v3: instead of failing make it more resilient
meta/classes/archiver.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index c19c770d11..b1d7aa0247 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -485,7 +485,8 @@ python do_unpack_and_patch() {
# Save the original source for creating the patches
if d.getVarFlag('ARCHIVER_MODE', 'diff') == '1':
src = d.getVar('S').rstrip('/')
-src_orig = '%s.orig' % src
+# append machine name to avoid races in multiconfig when using the
same TMPDIR
+src_orig = '%s.orig.%s' % (src, d.getVar('MACHINE'))
oe.path.copytree(src, src_orig)
if bb.data.inherits_class('dos2unix', d):
--
2.36.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#166784):
https://lists.openembedded.org/g/openembedded-core/message/166784
Mute This Topic: https://lists.openembedded.org/mt/91650171/21656
Group Owner: openembedded-core+ow...
