ODP: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
-09-20 03:32:03.0 +0200 netlink.c -rw-r--r-- 2 plobacz plobacz 1066 2009-04-29 16:37:13.0 +0200 netlink.h -rw-r--r-- 2 plobacz plobacz 4876 2015-07-29 00:32:35.0 +0200 proc.c -rw-r--r-- 2 plobacz plobacz 1066 2012-02-16 21:25:02.0 +0100 proc.h -rw-r--r-- 2 plobacz plobacz 5018 2022-09-16 03:24:42.0 +0200 sock.c -rw-r--r-- 2 plobacz plobacz 1198 2012-02-16 21:25:02.0 +0100 sock.h -rw-r--r-- 2 plobacz plobacz 3007 2016-01-27 17:13:18.0 +0100 ud_socket.c -rw-r--r-- 2 plobacz plobacz 352 2012-03-21 00:54:18.0 +0100 ud_socket.h This is rather unexpected, because by default tar is in posix format. I need to talk about it with tar people, but hell yeah finally we got it :-) BR Piotr Od: openembedded-core@lists.openembedded.org w imieniu użytkownika Piotr Łobacz via lists.openembedded.org Wysłane: wtorek, 1 sierpnia 2023 18:00 Do: Richard Purdie ; Martin Jansa DW: Joshua Watt ; Alexandre Belloni ; openembedded-core@lists.openembedded.org ; Alex Stewart Temat: ODP: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr That is what I was thinking for the first time, but by default tar is being run with posix, so, it is not possible that these commands are responsible for this... My guess is about do_package from package.bbclass and it calls this createCopy function which calls copyhardlinktree. And to my supprise there is also tar command, but it is written there: # Need to copy directories only with tar first since cp will error if two # writers try and create a directory at the same time and afterwards is this cp command with --preserve=xattrs and according to cp manual we can also preserve timestamps. Correct me if my thinking is wrong. I'm giving it a shot and will inform you about progress. BR Piotr Od: Richard Purdie Wysłane: wtorek, 1 sierpnia 2023 17:41 Do: Piotr Łobacz ; Martin Jansa DW: Joshua Watt ; Alexandre Belloni ; openembedded-core@lists.openembedded.org ; Alex Stewart Temat: Re: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr On Tue, 2023-08-01 at 15:16 +, Piotr Łobacz wrote: > Hi Richard, > you were right. I have made another test with wiping out whole sstate-cache > for acpid and rerunning the whole reproducible test. > It occurs that the package > /media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build/sstate-cache/a7/41/sstate:acpid:core2-64-poky-linux:2.0.34:r0:core2-64:11:a741afe7c0bfc218a844a546735ddc4dcaf578bff5d6a3762b1bdedf0819fc9b_package.tar.zst > was already containing wrong timestamps. > > > I'd guess you need to make the sstate archives of do_package store the > > acl and extended attribute data and when that is done, the millisecond > > timestamp data will probably be saved too? > > So I'm guessing that commands which are producing this > package_name_package.tar.zst are producing it in gnu format. Question is > where can I find it? > Is it being produced in this sstate.bbclass or elsewhere? https://git.yoctoproject.org/poky/tree/meta/classes-global/sstate.bbclass#n859 tar -I "$ZSTD" $OPT -f $TFILE * or tar -I "$ZSTD" $OPT --file=$TFILE --files-from=/dev/null Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#185368): https://lists.openembedded.org/g/openembedded-core/message/185368 Mute This Topic: https://lists.openembedded.org/mt/100497199/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
ODP: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
That is what I was thinking for the first time, but by default tar is being run with posix, so, it is not possible that these commands are responsible for this... My guess is about do_package from package.bbclass and it calls this createCopy function which calls copyhardlinktree. And to my supprise there is also tar command, but it is written there: # Need to copy directories only with tar first since cp will error if two # writers try and create a directory at the same time and afterwards is this cp command with --preserve=xattrs and according to cp manual we can also preserve timestamps. Correct me if my thinking is wrong. I'm giving it a shot and will inform you about progress. BR Piotr Od: Richard Purdie Wysłane: wtorek, 1 sierpnia 2023 17:41 Do: Piotr Łobacz ; Martin Jansa DW: Joshua Watt ; Alexandre Belloni ; openembedded-core@lists.openembedded.org ; Alex Stewart Temat: Re: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr On Tue, 2023-08-01 at 15:16 +, Piotr Łobacz wrote: > Hi Richard, > you were right. I have made another test with wiping out whole sstate-cache > for acpid and rerunning the whole reproducible test. > It occurs that the package > /media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build/sstate-cache/a7/41/sstate:acpid:core2-64-poky-linux:2.0.34:r0:core2-64:11:a741afe7c0bfc218a844a546735ddc4dcaf578bff5d6a3762b1bdedf0819fc9b_package.tar.zst > was already containing wrong timestamps. > > > I'd guess you need to make the sstate archives of do_package store the > > acl and extended attribute data and when that is done, the millisecond > > timestamp data will probably be saved too? > > So I'm guessing that commands which are producing this > package_name_package.tar.zst are producing it in gnu format. Question is > where can I find it? > Is it being produced in this sstate.bbclass or elsewhere? https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.yoctoproject.org%2Fpoky%2Ftree%2Fmeta%2Fclasses-global%2Fsstate.bbclass%23n859=05%7C01%7Cp.lobacz%40welotec.com%7C6ede1a2d1d444ff526c208db92a5cf36%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638265013099817702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=1JiMQn00kK9Kddz6%2FMhPVKThB3crvKNTz9%2FrTOt14hM%3D=0<https://git.yoctoproject.org/poky/tree/meta/classes-global/sstate.bbclass#n859> tar -I "$ZSTD" $OPT -f $TFILE * or tar -I "$ZSTD" $OPT --file=$TFILE --files-from=/dev/null Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#185253): https://lists.openembedded.org/g/openembedded-core/message/185253 Mute This Topic: https://lists.openembedded.org/mt/100486868/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
ODP: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
Hello again,
Funny thing, I have run the same revision as you in here
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3263
and I have different failures, but I have them:
File
"/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/meta/lib/oeqa/selftest/cases/reproducible.py",
line 327, in test_reproducible_builds
self.fail('\n'.join(fails))
AssertionError: The following ipk packages are different and not in exclusion
list:
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/acpid-src_2.0.34-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/avahi-src_0.8-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/bind-src_9.18.16-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/debugedit-src_5.0-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/flac-src_1.4.3-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-dev_1.6.6-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-ptest_1.6.6-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libjson-glib-1.0-src_1.6.6-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libmnl-src_1.0.5-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/libuuid-src_2.38.1-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/log4cplus-src_2.1.0-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/lttng-tools-src_2.13.9-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-numpy-src_1.25.1-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject-dev_3.44.1-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject-src_3.44.1-r0_core2-64.ipk
/media/plobacz/9582beb9-8774-4d4d-976f-63a6f1c87c13/poky-contrib/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/python3-pygobject_3.44.1-r0_core2-64.ipk
2023-07-29 00:18:31,843 - oe-selftest - INFO -
--
2023-07-29 00:18:31,843 - oe-selftest - INFO - Ran 2 tests in 51978.070s
2023-07-29 00:18:31,843 - oe-selftest - INFO - FAILED
2023-07-29 00:18:31,843 - oe-selftest - INFO - (failures=1)
2023-07-29 00:18:35,545 - oe-selftest - INFO - RESULTS:
2023-07-29 00:18:35,546 - oe-selftest - INFO - RESULTS -
reproducible.DiffoscopeTests.test_diffoscope: PASSED (35.32s)
2023-07-29 00:18:35,546 - oe-selftest - INFO - RESULTS -
reproducible.ReproducibleTests.test_reproducible_builds: FAILED (51940.60s)
2023-07-29 00:18:35,948 - oe-selftest - INFO - SUMMARY:
2023-07-29 00:18:35,949 - oe-selftest - INFO - oe-selftest () - Ran 2 tests in
51978.070s
2023-07-29 00:18:35,949 - oe-selftest - INFO - oe-selftest - FAIL - Required
tests failed (successes=1, skipped=0, failures=1, errors=0)
due to this tests result difference, my suspicion at the moment is only the
possibility of changing the format between gnu and posix depending on
DISTRO_FEATURES having acl and/or xattr.
I will force now tar format to have only posix format and run another test. We
will be in touch.
BR
Piotr
Od: Alexandre Belloni
Wysłane: czwartek, 27 lipca 2023 16:18
Do: Piotr Łobacz
DW: openembedded-core@lists.openembedded.org
; Alex Stewart
Temat: Re: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add
support for ACLs and xattr
Hello,
I sent this to the autobuilders and I got the same reproducibility
failure as earlier:
https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fautobuilder.yoctoproject.org%2Ftyphoon%2F%23%2Fbuilders%2F117%2Fbuilds%2F3263%2Fsteps%2F13%2Flogs%2Fstdio=05%7C01%7Cp.lobacz%40welotec.com%7C8fb759c0e1ab49b0c69408db8eac5547%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638260643038719160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=C6mYlLwOmYosyl1ZIS8vKQf2ailFqMqBXeamF%2FnSE7E%3D=0<https://autobuilder.yoctoproject.org/typho
ODP: ODP: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support for ACLs and xattr
Hi Alexandre,
> Hello,
>
> I sent this to the autobuilders and I got the same reproducibility
> failure as earlier:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/3263/steps/13/logs/stdio>data=05%7C01%7Cp.lobacz%40welotec.com%7C8fb759c0e1ab49b0c69408db8eac5547%7C25111a7f1d5a4c51a4ca7f8e44011b39%7C0%7C0%7C638260643038719160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=C6mYlLwOmYosyl1ZIS8vKQf2ailFqMqBXeamF%2FnSE7E%3D=0
>
> I'm not sure why you wouldn't see those.
Nor do I. I'm running it once again on our remote CI server, maybe I missed
something. I'll answer you when I will know something more.
BR
Piotr
On 26/07/2023 09:27:53+, Piotr Łobacz wrote:
>
> Hi Alexandre, Alex,
> Thx for supporting me, I have finally discovered the issue regarding this
> packages differences.
> It occurred that I was badly comparing two strings for tarformat comparison.
> Now it is fixed and should finally work.
>
> This has been additionally tested with oe-selftest -r reproducible.
>
> BR
> Piotr
>
> Od: Piotr Łobacz
> Wysłane: środa, 26 lipca 2023 11:22
> Do: openembedded-core@lists.openembedded.org
>
> DW: Piotr Łobacz
> Temat: [OE-Core][PATCH v11][master-next 1/5] package_ipk.bbclass: add support
> for ACLs and xattr
>
> Extend OPKGBUILDCMD variable, with additional parameters, depending
> on target distro features, in order to support ACLs and xattr.
>
> With fix pushed to the opkg-devel:
> https://groups.google.com/g/opkg-devel/c/dYNHrLjDwg8
> opkg-build is able to create tar archives with ACLs and xattr.
>
> Signed-off-by: Piotr Łobacz
> ---
> meta/classes-global/package_ipk.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-global/package_ipk.bbclass
> b/meta/classes-global/package_ipk.bbclass
> index b4b7bc9ac2..a0f106e4ad 100644
> --- a/meta/classes-global/package_ipk.bbclass
> +++ b/meta/classes-global/package_ipk.bbclass
> @@ -15,7 +15,7 @@ IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"
> PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"
>
> # Program to be used to build opkg packages
> -OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"'
> +OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"
> ${@bb.utils.contains('DISTRO_FEATURES', 'acl', '-A', '', d)}
> ${@bb.utils.contains('DISTRO_FEATURES', 'xattr', '-X', '', d)}'
>
> OPKG_ARGS += "--force_postinstall --prefer-arch-to-version"
> OPKG_ARGS += "${@['',
> '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}"
> --
> 2.34.1
>
>
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com/
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#184963):
https://lists.openembedded.org/g/openembedded-core/message/184963
Mute This Topic: https://lists.openembedded.org/mt/100392767/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
