Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs
On 2017-09-15 20:20, Martin Hundebøll wrote: On 2017-09-15 14:16, Tom Rini wrote: This commit has resulted in more or less persistent basehash issues for my cpio.gz.u-boot image: ERROR: When reparsing /home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio, the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to 8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic and this needs to be fixed. The error occurs repeatedly when doing `bitbake -c clean rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes the error go away. Also, the issue only appears for cpio.gz.u-boot image types, but not for cpio.gz I am far from an expert in bitbake signature handling, so please let me know what to do, to debug this further. Can you let me know what layers and machine I need to reproduce this? I'll take a look, thanks! I am able to reproduce with the following: - Ubuntu 16.04 based on the attached Dockerfile - A clean clone of poky (commit 072430b9b3) - meta, meta-poky, and meta-yocto-bsp (see build-config.log) - beaglebone machine and gpio.gz.u-boot image type (see local.conf) Sorry, this seems to be caused by your earlier patch: 88eec32150 ('image.bbclass: Correct chaining compression support'). My first conclusion was probably caused by caching playing me a trick. I'll reply to that patch instead. // Martin -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs
Hi Tom, On 2017-09-15 14:16, Tom Rini wrote: This commit has resulted in more or less persistent basehash issues for my cpio.gz.u-boot image: ERROR: When reparsing /home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio, the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to 8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic and this needs to be fixed. The error occurs repeatedly when doing `bitbake -c clean rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes the error go away. Also, the issue only appears for cpio.gz.u-boot image types, but not for cpio.gz I am far from an expert in bitbake signature handling, so please let me know what to do, to debug this further. Can you let me know what layers and machine I need to reproduce this? I'll take a look, thanks! I am able to reproduce with the following: - Ubuntu 16.04 based on the attached Dockerfile - A clean clone of poky (commit 072430b9b3) - meta, meta-poky, and meta-yocto-bsp (see build-config.log) - beaglebone machine and gpio.gz.u-boot image type (see local.conf) // Martin FROM ubuntu:16.04 MAINTAINER Martin HundebøllENV DEBIAN_FRONTEND=noninteractive RUN apt-get update RUN apt-get install -y python python3 build-essential locales RUN apt-get install -y chrpath cpio diffstat gawk git texinfo wget iputils-ping openssh-client RUN apt-get install -y zsh vim less RUN locale-gen en_US.UTF-8 ENV LANG=en_US.UTF-8 RUN groupadd admin; \ useradd -m -p $(openssl passwd -crypt foobar) -s /bin/zsh -G admin mnhu; USER mnhu WORKDIR /home/mnhu CMD ["/bin/zsh"] # # This file is your local configuration file and is where all local user settings # are placed. The comments in this file give some guide to the options a new user # to the system might want to change but pretty much any configuration option can # be set in this file. More adventurous users can look at local.conf.extended # which contains other examples of configuration which can be placed in this file # but new users likely won't need any of them initially. # # Lines starting with the '#' character are commented out and in some cases the # default values are provided as comments to show people example syntax. Enabling # the option is a question of removing the # character and making any change to the # variable as required. # # Machine Selection # # You need to select a specific machine to target the build with. There are a selection # of emulated machines available which can boot and run in the QEMU emulator: # #MACHINE ?= "qemuarm" #MACHINE ?= "qemuarm64" #MACHINE ?= "qemumips" #MACHINE ?= "qemumips64" #MACHINE ?= "qemuppc" #MACHINE ?= "qemux86" #MACHINE ?= "qemux86-64" # # There are also the following hardware board target machines included for # demonstration purposes: # #MACHINE ?= "beaglebone" #MACHINE ?= "genericx86" #MACHINE ?= "genericx86-64" #MACHINE ?= "mpc8315e-rdb" #MACHINE ?= "edgerouter" # # This sets the default machine to be qemux86 if no other machine is selected: MACHINE ??= "beaglebone" # # Where to place downloads # # During a first build the system will download many different source code tarballs # from various upstream projects. This can take a while, particularly if your network # connection is slow. These are all stored in DL_DIR. When wiping and rebuilding you # can preserve this directory to speed up this part of subsequent builds. This directory # is safe to share between multiple builds on the same machine too. # # The default is a downloads directory under TOPDIR which is the build directory. # #DL_DIR ?= "${TOPDIR}/downloads" # # Where to place shared-state files # # BitBake has the capability to accelerate builds based on previously built output. # This is done using "shared state" files which can be thought of as cache objects # and this option determines where those files are placed. # # You can wipe out TMPDIR leaving this directory intact and the build would regenerate # from these files if no changes were made to the configuration. If changes were made # to the configuration, only shared state files where the state was still valid would # be used (done using checksums). # # The default is a sstate-cache directory under TOPDIR. # #SSTATE_DIR ?= "${TOPDIR}/sstate-cache" # # Where to place the build output # # This option specifies where the bulk of the building work should be done and # where BitBake should place its temporary files and output. Keep in mind that # this includes the extraction and compilation of many applications and the toolchain # which can use Gigabytes of hard disk space. # # The default is a tmp directory under TOPDIR. # #TMPDIR = "${TOPDIR}/tmp" # # Default policy config # # The distribution setting controls which policy settings are used as defaults. # The default value is fine for general Yocto project use, at least initially. # Ultimately when creating custom policy, people will likely end up
Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs
On Fri, Sep 15, 2017 at 09:41:36AM +0200, Martin Hundebøll wrote: > > > On 2017-07-25 21:58, Tom Rini wrote: > >When we have more than one CONVERSION_CMD being used, for example > >ext4.gz.sha256sum we will see errors about "metadata is not > >deterministic". This is because we do not have a stable order of > >intermediate files that will be removed in the generated shell command. > >We fix this by calling sorted() on the set of rm_tmp_images so that we > >will have a stable hash again. > > > >Cc: Patrick Ohly> >Signed-off-by: Tom Rini > >--- > > meta/classes/image.bbclass | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > >diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > >index b095bca..dee17ae 100644 > >--- a/meta/classes/image.bbclass > >+++ b/meta/classes/image.bbclass > >@@ -474,7 +474,7 @@ python () { > > # Clean up after applying all conversion commands. Some of them > > might > > # use the same input, therefore we cannot delete sooner without > > applying > > # some complex dependency analysis. > >-for image in rm_tmp_images: > >+for image in sorted(rm_tmp_images): > > cmds.append("\trm " + image) > > after = 'do_image' > > > > This commit has resulted in more or less persistent basehash issues > for my cpio.gz.u-boot image: > > ERROR: When reparsing > /home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio, > the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to > 8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic > and this needs to be fixed. > > The error occurs repeatedly when doing `bitbake -c clean > rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes > the error go away. > > Also, the issue only appears for cpio.gz.u-boot image types, but not > for cpio.gz > > I am far from an expert in bitbake signature handling, so please let > me know what to do, to debug this further. Can you let me know what layers and machine I need to reproduce this? I'll take a look, thanks! -- Tom signature.asc Description: Digital signature -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs
On Tue, 2017-07-25 at 15:58 -0400, Tom Rini wrote: > When we have more than one CONVERSION_CMD being used, for example > ext4.gz.sha256sum we will see errors about "metadata is not > deterministic". This is because we do not have a stable order of > intermediate files that will be removed in the generated shell > command. > We fix this by calling sorted() on the set of rm_tmp_images so that > we > will have a stable hash again. > > Cc: Patrick Ohly> Signed-off-by: Tom Rini Acked-by: Patrick Ohly -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs
When we have more than one CONVERSION_CMD being used, for example ext4.gz.sha256sum we will see errors about "metadata is not deterministic". This is because we do not have a stable order of intermediate files that will be removed in the generated shell command. We fix this by calling sorted() on the set of rm_tmp_images so that we will have a stable hash again. Cc: Patrick OhlySigned-off-by: Tom Rini --- meta/classes/image.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index b095bca..dee17ae 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -474,7 +474,7 @@ python () { # Clean up after applying all conversion commands. Some of them might # use the same input, therefore we cannot delete sooner without applying # some complex dependency analysis. -for image in rm_tmp_images: +for image in sorted(rm_tmp_images): cmds.append("\trm " + image) after = 'do_image' -- 2.1.4 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core