Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs

[Martin Hundebøll]

On 2017-09-15 20:20, Martin Hundebøll wrote:

On 2017-09-15 14:16, Tom Rini wrote:

This commit has resulted in more or less persistent basehash issues
for my cpio.gz.u-boot image:

ERROR: When reparsing 
/home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio, 


the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to
8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic
and this needs to be fixed.

The error occurs repeatedly when doing `bitbake -c clean
rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes
the error go away.

Also, the issue only appears for cpio.gz.u-boot image types, but not
for cpio.gz

I am far from an expert in bitbake signature handling, so please let
me know what to do, to debug this further.

Can you let me know what layers and machine I need to reproduce this?
I'll take a look, thanks!


I am able to reproduce with the following:

   - Ubuntu 16.04 based on the attached Dockerfile
   - A clean clone of poky (commit 072430b9b3)
   - meta, meta-poky, and meta-yocto-bsp (see build-config.log)
   - beaglebone machine and gpio.gz.u-boot image type (see local.conf)


Sorry, this seems to be caused by your earlier patch: 88eec32150 
('image.bbclass: Correct chaining compression support'). My first 
conclusion was probably caused by caching playing me a trick.


I'll reply to that patch instead.

// Martin
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs

[Martin Hundebøll]

Hi Tom,

On 2017-09-15 14:16, Tom Rini wrote:

This commit has resulted in more or less persistent basehash issues
for my cpio.gz.u-boot image:

ERROR: When reparsing 
/home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio,
the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to
8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic
and this needs to be fixed.

The error occurs repeatedly when doing `bitbake -c clean
rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes
the error go away.

Also, the issue only appears for cpio.gz.u-boot image types, but not
for cpio.gz

I am far from an expert in bitbake signature handling, so please let
me know what to do, to debug this further.

Can you let me know what layers and machine I need to reproduce this?
I'll take a look, thanks!


I am able to reproduce with the following:

  - Ubuntu 16.04 based on the attached Dockerfile
  - A clean clone of poky (commit 072430b9b3)
  - meta, meta-poky, and meta-yocto-bsp (see build-config.log)
  - beaglebone machine and gpio.gz.u-boot image type (see local.conf)

// Martin

FROM ubuntu:16.04
MAINTAINER Martin Hundebøll 

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y python python3 build-essential locales
RUN apt-get install -y chrpath cpio diffstat gawk git texinfo wget iputils-ping 
openssh-client
RUN apt-get install -y zsh vim less
RUN locale-gen en_US.UTF-8
ENV LANG=en_US.UTF-8

RUN groupadd admin; \
useradd -m -p $(openssl passwd -crypt foobar) -s /bin/zsh -G admin mnhu;
USER mnhu
WORKDIR /home/mnhu
CMD ["/bin/zsh"]

#
# This file is your local configuration file and is where all local user 
settings
# are placed. The comments in this file give some guide to the options a new 
user
# to the system might want to change but pretty much any configuration option 
can
# be set in this file. More adventurous users can look at local.conf.extended
# which contains other examples of configuration which can be placed in this 
file
# but new users likely won't need any of them initially.
#
# Lines starting with the '#' character are commented out and in some cases the
# default values are provided as comments to show people example syntax. 
Enabling
# the option is a question of removing the # character and making any change to 
the
# variable as required.

#
# Machine Selection
#
# You need to select a specific machine to target the build with. There are a 
selection
# of emulated machines available which can boot and run in the QEMU emulator:
#
#MACHINE ?= "qemuarm"
#MACHINE ?= "qemuarm64"
#MACHINE ?= "qemumips"
#MACHINE ?= "qemumips64"
#MACHINE ?= "qemuppc"
#MACHINE ?= "qemux86"
#MACHINE ?= "qemux86-64"
#
# There are also the following hardware board target machines included for 
# demonstration purposes:
#
#MACHINE ?= "beaglebone"
#MACHINE ?= "genericx86"
#MACHINE ?= "genericx86-64"
#MACHINE ?= "mpc8315e-rdb"
#MACHINE ?= "edgerouter"
#
# This sets the default machine to be qemux86 if no other machine is selected:
MACHINE ??= "beaglebone"

#
# Where to place downloads
#
# During a first build the system will download many different source code 
tarballs
# from various upstream projects. This can take a while, particularly if your 
network
# connection is slow. These are all stored in DL_DIR. When wiping and 
rebuilding you
# can preserve this directory to speed up this part of subsequent builds. This 
directory
# is safe to share between multiple builds on the same machine too.
#
# The default is a downloads directory under TOPDIR which is the build 
directory.
#
#DL_DIR ?= "${TOPDIR}/downloads"

#
# Where to place shared-state files
#
# BitBake has the capability to accelerate builds based on previously built 
output.
# This is done using "shared state" files which can be thought of as cache 
objects
# and this option determines where those files are placed.
#
# You can wipe out TMPDIR leaving this directory intact and the build would 
regenerate
# from these files if no changes were made to the configuration. If changes 
were made
# to the configuration, only shared state files where the state was still valid 
would
# be used (done using checksums).
#
# The default is a sstate-cache directory under TOPDIR.
#
#SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

#
# Where to place the build output
#
# This option specifies where the bulk of the building work should be done and
# where BitBake should place its temporary files and output. Keep in mind that
# this includes the extraction and compilation of many applications and the 
toolchain
# which can use Gigabytes of hard disk space.
#
# The default is a tmp directory under TOPDIR.
#
#TMPDIR = "${TOPDIR}/tmp"

#
# Default policy config
#
# The distribution setting controls which policy settings are used as defaults.
# The default value is fine for general Yocto project use, at least initially.
# Ultimately when creating custom policy, people will likely end up 

Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs

[Tom Rini]

On Fri, Sep 15, 2017 at 09:41:36AM +0200, Martin Hundebøll wrote:
> 
> 
> On 2017-07-25 21:58, Tom Rini wrote:
> >When we have more than one CONVERSION_CMD being used, for example
> >ext4.gz.sha256sum we will see errors about "metadata is not
> >deterministic".  This is because we do not have a stable order of
> >intermediate files that will be removed in the generated shell command.
> >We fix this by calling sorted() on the set of rm_tmp_images so that we
> >will have a stable hash again.
> >
> >Cc: Patrick Ohly 
> >Signed-off-by: Tom Rini 
> >---
> >  meta/classes/image.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> >index b095bca..dee17ae 100644
> >--- a/meta/classes/image.bbclass
> >+++ b/meta/classes/image.bbclass
> >@@ -474,7 +474,7 @@ python () {
> >  # Clean up after applying all conversion commands. Some of them 
> > might
> >  # use the same input, therefore we cannot delete sooner without 
> > applying
> >  # some complex dependency analysis.
> >-for image in rm_tmp_images:
> >+for image in sorted(rm_tmp_images):
> >  cmds.append("\trm " + image)
> >  after = 'do_image'
> >
> 
> This commit has resulted in more or less persistent basehash issues
> for my cpio.gz.u-boot image:
> 
> ERROR: When reparsing 
> /home/mnhu/projects/pil/yocto/meta-pil/recipes-filesystems/rescue-rootfs.bb.do_image_cpio,
> the basehash value changed from ac9907bbb8ee2ee6d9ca90f7a7e265f5 to
> 8c7b717add6b1d1ba6ff761ebfdedf01. The metadata is not deterministic
> and this needs to be fixed.
> 
> The error occurs repeatedly when doing `bitbake -c clean
> rescue-rootfs && bitbake rescue-rootfs`. Reverting the commit makes
> the error go away.
> 
> Also, the issue only appears for cpio.gz.u-boot image types, but not
> for cpio.gz
> 
> I am far from an expert in bitbake signature handling, so please let
> me know what to do, to debug this further.

Can you let me know what layers and machine I need to reproduce this?
I'll take a look, thanks!

-- 
Tom


signature.asc
Description: Digital signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] image: Fix "metadata is not deterministic" when chaining 2+ CONVERSION_CMDs

[Patrick Ohly]

On Tue, 2017-07-25 at 15:58 -0400, Tom Rini wrote:
> When we have more than one CONVERSION_CMD being used, for example
> ext4.gz.sha256sum we will see errors about "metadata is not
> deterministic".  This is because we do not have a stable order of
> intermediate files that will be removed in the generated shell
> command.
> We fix this by calling sorted() on the set of rm_tmp_images so that
> we
> will have a stable hash again.
> 
> Cc: Patrick Ohly 
> Signed-off-by: Tom Rini 

Acked-by: Patrick Ohly 

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core