Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Fri, Oct 12, 2018 at 12:21:14PM +, Baeuerle, Florian wrote:
> Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > > Inject fake timestamps into the build via faketime to produce the exact
> > > same binaries.
> > >
> > > Signed-off-by: Florian Bäuerle
> > > ---
> > > platforms/barebox.in | 10 ++
> > > rules/barebox.make | 16 +++-
> > > 2 files changed, 25 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > > index 645366e93..5bc2f08b6 100644
> > > --- a/platforms/barebox.in
> > > +++ b/platforms/barebox.in
> > > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > > prompt "barebox "
> > > bool
> > > help
> > > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > > Directories and files to add to the builtin barebox environment.
> > > Relative paths must be relative to the platformconfigdir.
> > >
> > > +config BAREBOX_REPRODUCIBLE
> > > + prompt "build reproducibly"
> > > + bool
> > > + default n
> > > + help
> > > + With this, fake timestamps are injected to the barebox build, so the
> > > + resulting barebox images are identical when the sources haven't been
> > > + changed.
> > > +
> > > config BAREBOX_BAREBOXENV
> > > prompt "install 'bareboxenv'"
> > > bool
> > > diff --git a/rules/barebox.make b/rules/barebox.make
> > > index a6ec3c394..ad160b3ba 100644
> > > --- a/rules/barebox.make
> > > +++ b/rules/barebox.make
> > > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > > KCONFIG_NOTIMESTAMP=1 \
> > > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> > >
> > > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > > +BAREBOX_FAKE_TIMESTAMP := \
> > > + $(shell echo $(BAREBOX_VERSION) | \
> > > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> > > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > > + KBUILD_BUILD_VERSION="0" \
> > > + KBUILD_BUILD_USER="ptxdist" \
> > > + KBUILD_BUILD_HOST="ptxdist"
> >
> > What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> > hammer that can cause other problems. I'd like to understand what happens
> > here first.
>
> According to my observations, only the compression. A few compressors are
> capable of not including timestamps (e.g. gzip: --no-name), but others are not
> (lzop).
>
> To fix that, there are a few options I do not like:
>
> 1. fix the in the barebox build system and implement ugly hacks so this works
> for lzop (overwriting an lzo header field) and other compressors that do not
> support this out of the box.
>
> 2. fix this in ptxdist, probably by doing the same ugly stuff before copying
> images to IMAGEDIR
>
> 3. fix the problems in lzop and other compressors that do not support faking
> or
> skipping those header fields.
>
> The thirds option is still the best, but it will take ages to reach
> distributions.
I've looked at this a bit. For gzip, barebox already uses '-n', so that
should work correctly. Most other compressors don't add a timestamp, at
least in my quick tests. What remains is lzop.
We already have a BAREBOX_NEEDS_HOST_LZOP option, so the only thing needed
is a patch. Using SOURCE_DATE_EPOCH for mtime_low/mtime_high if available
(instead of the file time) should be trivial.
Michael
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Tue, Oct 16, 2018 at 02:27:05PM +, Baeuerle, Florian wrote: > Am Dienstag, den 16.10.2018, 14:38 +0200 schrieb Michael Olbrich: > > On Tue, Oct 16, 2018 at 10:57:20AM +, Baeuerle, Florian wrote: > > > Am Dienstag, den 16.10.2018, 12:09 +0200 schrieb Michael Olbrich: > > > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote: > > > > > Inject fake timestamps into the build via faketime to produce the > > > > > exact > > > > > same binaries. > > > > > > > > I'd like to see this patch split in two. > > > > One to handle the KBUILD_BUILD_ stuff. That should be done like the > > > > kernel > > > > with the symlink magic, but otherwise unconditionally (for now). > > > > Use SOURCE_DATE_EPOCH for the timestamp. I want to change the kernel to > > > > use > > > > this too. > > > > > > SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus > > > the > > > timestamp changes with every ptxdist release. I'd prefer to *really* get > > > the > > > same binary when the source code was not changed at all. A ptxdist version > > > bump > > > has a high potential of not touching anything barebox related. > > > > > > I also do not like mangling the barebox version into a timestamp, but I > > > couldn't > > > come up with any better idea. > > > > You can easily overwrite that in your BSP and I'm wouldn't mind changing > > that in general. The PTXdist release was just a convenient 'date' to use. > > I'd like to have something that is global to the BSP. Do you have any > > sugestions? > > Also I'd prefer to use something BSP-global, but I couldn't find anything > suitable: > > PTXCONF_PROJECT_VERSION - not guaranteed to be a date > PTXCONF_PLATFORM_VERSION - same here > > Maybe ptxdist could offer using one of those as a fake timestamp, if the > appropriate format is used. I'd expect that these change as well. Hmmm, maybe make it configurable - ptxdist release: what we have now - toolchain release: if the compiler changes, then the result will be different anyways. - manual 'date' string: any use-case for this? Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Tue, Oct 16, 2018 at 07:15:37PM +0200, Alexander Dahl wrote: > On Tue, Oct 16, 2018 at 10:57:20AM +, Baeuerle, Florian wrote: > > SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus > > the > > timestamp changes with every ptxdist release. I'd prefer to *really* get the > > same binary when the source code was not changed at all. A ptxdist version > > bump > > has a high potential of not touching anything barebox related. > > > > I also do not like mangling the barebox version into a timestamp, but I > > couldn't > > come up with any better idea. > > If I understood correctly, you want some timestamp which does not > change, as long as barebox is not touched. Why not base it on the > barebox version then? What would be the problem with that? That's what his patch does. I'd like to have one timestamp for all of PTXdist. Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Tue, Oct 16, 2018 at 02:11:52PM +, Baeuerle, Florian wrote: > Am Dienstag, den 16.10.2018, 14:51 +0200 schrieb Michael Olbrich: > > On Tue, Oct 16, 2018 at 11:06:19AM +, Baeuerle, Florian wrote: > > > Am Dienstag, den 16.10.2018, 10:59 +0200 schrieb Michael Olbrich: > > > > On Fri, Oct 12, 2018 at 12:28:19PM +, Baeuerle, Florian wrote: > > > > > It's an opt-in hammer at least. One annoyance that I observed is, that > > > > > menuconfig barebox also does build host-faketime and a bunch of other > > > > > things > > > > > that are not a dependency of the host-faketime package. > > > > > > > > That's the usual early build stuff, right? host-pkg-config and things > > > > like > > > > that. > > > > > > > > > > I guess so. It's host-chrpath, host-pkg-config, kernel-header, host-attr, > > > host- > > > autotools-*, host-acl, host-fakeroot, host-m4 to name a few. > > > > Hmmm, I applied the patches and did some quick checks, and I'm not seeing > > the issue. If you run ptxdist with '-d' you might get a hint where the > > dependency is coming from. > > It's not an issue, the same happens without the patches. I just never noticed > it. The dependencies come from base.install. So the question is, why does barebox_menuconfig depend on base.install. Probably via barebox.extract but that shouldn't happen either. Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Hello Florian, On Tue, Oct 16, 2018 at 10:57:20AM +, Baeuerle, Florian wrote: > SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus the > timestamp changes with every ptxdist release. I'd prefer to *really* get the > same binary when the source code was not changed at all. A ptxdist version > bump > has a high potential of not touching anything barebox related. > > I also do not like mangling the barebox version into a timestamp, but I > couldn't > come up with any better idea. If I understood correctly, you want some timestamp which does not change, as long as barebox is not touched. Why not base it on the barebox version then? What would be the problem with that? Greets Alex -- /"\ ASCII RIBBON | »With the first link, the chain is forged. The first \ / CAMPAIGN | speech censured, the first thought forbidden, the X AGAINST | first freedom denied, chains us all irrevocably.« / \ HTML MAIL| (Jean-Luc Picard, quoting Judge Aaron Satie) signature.asc Description: PGP signature ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Am Dienstag, den 16.10.2018, 14:38 +0200 schrieb Michael Olbrich: > On Tue, Oct 16, 2018 at 10:57:20AM +, Baeuerle, Florian wrote: > > Hi Michael, > > > > Am Dienstag, den 16.10.2018, 12:09 +0200 schrieb Michael Olbrich: > > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote: > > > > Inject fake timestamps into the build via faketime to produce the exact > > > > same binaries. > > > > > > I'd like to see this patch split in two. > > > One to handle the KBUILD_BUILD_ stuff. That should be done like the kernel > > > with the symlink magic, but otherwise unconditionally (for now). > > > Use SOURCE_DATE_EPOCH for the timestamp. I want to change the kernel to > > > use > > > this too. > > > > SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus > > the > > timestamp changes with every ptxdist release. I'd prefer to *really* get the > > same binary when the source code was not changed at all. A ptxdist version > > bump > > has a high potential of not touching anything barebox related. > > > > I also do not like mangling the barebox version into a timestamp, but I > > couldn't > > come up with any better idea. > > You can easily overwrite that in your BSP and I'm wouldn't mind changing > that in general. The PTXdist release was just a convenient 'date' to use. > I'd like to have something that is global to the BSP. Do you have any > sugestions? Also I'd prefer to use something BSP-global, but I couldn't find anything suitable: PTXCONF_PROJECT_VERSION - not guaranteed to be a date PTXCONF_PLATFORM_VERSION - same here Maybe ptxdist could offer using one of those as a fake timestamp, if the appropriate format is used. - Florian ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Am Dienstag, den 16.10.2018, 14:51 +0200 schrieb Michael Olbrich:
> On Tue, Oct 16, 2018 at 11:06:19AM +, Baeuerle, Florian wrote:
> > Am Dienstag, den 16.10.2018, 10:59 +0200 schrieb Michael Olbrich:
> > > On Fri, Oct 12, 2018 at 12:28:19PM +, Baeuerle, Florian wrote:
> > > > Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> > > > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > > > > > Inject fake timestamps into the build via faketime to produce the
> > > > > > exact
> > > > > > same binaries.
> > > > > >
> > > > > > Signed-off-by: Florian Bäuerle
> > > > > > ---
> > > > > > platforms/barebox.in | 10 ++
> > > > > > rules/barebox.make | 16 +++-
> > > > > > 2 files changed, 25 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > > > > > index 645366e93..5bc2f08b6 100644
> > > > > > --- a/platforms/barebox.in
> > > > > > +++ b/platforms/barebox.in
> > > > > > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > > > > > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > > > > > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > > > > > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > > > > > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > > > > > prompt "barebox "
> > > > > > bool
> > > > > > help
> > > > > > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > > > > > Directories and files to add to the builtin barebox
> > > > > > environment.
> > > > > > Relative paths must be relative to the platformconfigdir.
> > > > > >
> > > > > > +config BAREBOX_REPRODUCIBLE
> > > > > > + prompt "build reproducibly"
> > > > > > + bool
> > > > > > + default n
> > > > > > + help
> > > > > > + With this, fake timestamps are injected to the barebox build,
> > > > > > so the
> > > > > > + resulting barebox images are identical when the sources
> > > > > > haven't been
> > > > > > + changed.
> > > > > > +
> > > > > > config BAREBOX_BAREBOXENV
> > > > > > prompt "install 'bareboxenv'"
> > > > > > bool
> > > > > > diff --git a/rules/barebox.make b/rules/barebox.make
> > > > > > index a6ec3c394..ad160b3ba 100644
> > > > > > --- a/rules/barebox.make
> > > > > > +++ b/rules/barebox.make
> > > > > > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > > > > > KCONFIG_NOTIMESTAMP=1 \
> > > > > > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> > > > > >
> > > > > > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > > > > > +BAREBOX_FAKE_TIMESTAMP := \
> > > > > > + $(shell echo $(BAREBOX_VERSION) | \
> > > > > > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01
> > > > > > 00:00:00/")
> > > > > > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > > > > > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > > > > > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > > > > > + KBUILD_BUILD_VERSION="0" \
> > > > > > + KBUILD_BUILD_USER="ptxdist" \
> > > > > > + KBUILD_BUILD_HOST="ptxdist"
> > > >
> > > > For the sake of completeness: the _USER and _HOST variables are *not
> > > > yet*
> > > > incorporated into the resulting binaries by barebox (at least with my
> > > > configuration).
> > >
> > > Hmm, then something is wrong here. scripts/mkcompile_h has always used
> > > these variables.
> >
> > Yes, it generates #defines, but apparently these are not used anywhere in
> > the
> > barebox source tree, thus they do not end up in the binary.
>
> Ok, not used as in no user/host in the binary at all. That's ok then.
>
> > > > > What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> > > > > hammer that can cause other problems. I'd like to understand what
> > > > > happens
> > > > > here first.
> > > >
> > > > It's an opt-in hammer at least. One annoyance that I observed is, that
> > > > menuconfig barebox also does build host-faketime and a bunch of other
> > > > things
> > > > that are not a dependency of the host-faketime package.
> > >
> > > That's the usual early build stuff, right? host-pkg-config and things like
> > > that.
> > >
> >
> > I guess so. It's host-chrpath, host-pkg-config, kernel-header, host-attr,
> > host-
> > autotools-*, host-acl, host-fakeroot, host-m4 to name a few.
>
> Hmmm, I applied the patches and did some quick checks, and I'm not seeing
> the issue. If you run ptxdist with '-d' you might get a hint where the
> dependency is coming from.
It's not an issue, the same happens without the patches. I just never noticed
it. The dependencies come from base.install.
- Florian
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Tue, Oct 16, 2018 at 11:06:19AM +, Baeuerle, Florian wrote:
> Am Dienstag, den 16.10.2018, 10:59 +0200 schrieb Michael Olbrich:
> > On Fri, Oct 12, 2018 at 12:28:19PM +, Baeuerle, Florian wrote:
> > > Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> > > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > > > > Inject fake timestamps into the build via faketime to produce the
> > > > > exact
> > > > > same binaries.
> > > > >
> > > > > Signed-off-by: Florian Bäuerle
> > > > > ---
> > > > > platforms/barebox.in | 10 ++
> > > > > rules/barebox.make | 16 +++-
> > > > > 2 files changed, 25 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > > > > index 645366e93..5bc2f08b6 100644
> > > > > --- a/platforms/barebox.in
> > > > > +++ b/platforms/barebox.in
> > > > > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > > > > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > > > > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > > > > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > > > > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > > > > prompt "barebox "
> > > > > bool
> > > > > help
> > > > > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > > > > Directories and files to add to the builtin barebox
> > > > > environment.
> > > > > Relative paths must be relative to the platformconfigdir.
> > > > >
> > > > > +config BAREBOX_REPRODUCIBLE
> > > > > + prompt "build reproducibly"
> > > > > + bool
> > > > > + default n
> > > > > + help
> > > > > + With this, fake timestamps are injected to the barebox build,
> > > > > so the
> > > > > + resulting barebox images are identical when the sources
> > > > > haven't been
> > > > > + changed.
> > > > > +
> > > > > config BAREBOX_BAREBOXENV
> > > > > prompt "install 'bareboxenv'"
> > > > > bool
> > > > > diff --git a/rules/barebox.make b/rules/barebox.make
> > > > > index a6ec3c394..ad160b3ba 100644
> > > > > --- a/rules/barebox.make
> > > > > +++ b/rules/barebox.make
> > > > > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > > > > KCONFIG_NOTIMESTAMP=1 \
> > > > > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> > > > >
> > > > > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > > > > +BAREBOX_FAKE_TIMESTAMP := \
> > > > > + $(shell echo $(BAREBOX_VERSION) | \
> > > > > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01
> > > > > 00:00:00/")
> > > > > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > > > > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > > > > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > > > > + KBUILD_BUILD_VERSION="0" \
> > > > > + KBUILD_BUILD_USER="ptxdist" \
> > > > > + KBUILD_BUILD_HOST="ptxdist"
> > >
> > > For the sake of completeness: the _USER and _HOST variables are *not yet*
> > > incorporated into the resulting binaries by barebox (at least with my
> > > configuration).
> >
> > Hmm, then something is wrong here. scripts/mkcompile_h has always used
> > these variables.
>
> Yes, it generates #defines, but apparently these are not used anywhere in the
> barebox source tree, thus they do not end up in the binary.
Ok, not used as in no user/host in the binary at all. That's ok then.
> > > > What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> > > > hammer that can cause other problems. I'd like to understand what
> > > > happens
> > > > here first.
> > >
> > > It's an opt-in hammer at least. One annoyance that I observed is, that
> > > menuconfig barebox also does build host-faketime and a bunch of other
> > > things
> > > that are not a dependency of the host-faketime package.
> >
> > That's the usual early build stuff, right? host-pkg-config and things like
> > that.
> >
>
> I guess so. It's host-chrpath, host-pkg-config, kernel-header, host-attr,
> host-
> autotools-*, host-acl, host-fakeroot, host-m4 to name a few.
Hmmm, I applied the patches and did some quick checks, and I'm not seeing
the issue. If you run ptxdist with '-d' you might get a hint where the
dependency is coming from.
Michael
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Tue, Oct 16, 2018 at 10:57:20AM +, Baeuerle, Florian wrote: > Hi Michael, > > Am Dienstag, den 16.10.2018, 12:09 +0200 schrieb Michael Olbrich: > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote: > > > Inject fake timestamps into the build via faketime to produce the exact > > > same binaries. > > > > I'd like to see this patch split in two. > > One to handle the KBUILD_BUILD_ stuff. That should be done like the kernel > > with the symlink magic, but otherwise unconditionally (for now). > > Use SOURCE_DATE_EPOCH for the timestamp. I want to change the kernel to use > > this too. > > SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus the > timestamp changes with every ptxdist release. I'd prefer to *really* get the > same binary when the source code was not changed at all. A ptxdist version > bump > has a high potential of not touching anything barebox related. > > I also do not like mangling the barebox version into a timestamp, but I > couldn't > come up with any better idea. You can easily overwrite that in your BSP and I'm wouldn't mind changing that in general. The PTXdist release was just a convenient 'date' to use. I'd like to have something that is global to the BSP. Do you have any sugestions? > > Another patch for the rest. I'm not quite sure how I want to handle this. > > My current plan is to have a setup option: Developers can use it to disable > > reproducible builds to see stuff like the build timestamps etc. > > CI and release builds have it enabled by default. I currently don't have > > any reason do disable reproducibility for releases. > > > > What do you think? > > My main use case for reproducible builds is RAUC. Since bootloader updates are > especially hairy on some platforms, I'm happy about having the ability to > spare > a few of them (while retaining an update mechanism that ensures a persistent > state of the system *and* without using prebuilt bootloaders). > > However, I do not mind if ptxdist by default builds more packages > reproducibly. Updates are certainly one use-case. For me it's also useful to develop PTXdist. I can compare ipkgs from before and after some refactoring more easily. Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ ptxdist mailing list ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Am Dienstag, den 16.10.2018, 10:59 +0200 schrieb Michael Olbrich:
> On Fri, Oct 12, 2018 at 12:28:19PM +, Baeuerle, Florian wrote:
> > Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> > > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > > > Inject fake timestamps into the build via faketime to produce the exact
> > > > same binaries.
> > > >
> > > > Signed-off-by: Florian Bäuerle
> > > > ---
> > > > platforms/barebox.in | 10 ++
> > > > rules/barebox.make | 16 +++-
> > > > 2 files changed, 25 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > > > index 645366e93..5bc2f08b6 100644
> > > > --- a/platforms/barebox.in
> > > > +++ b/platforms/barebox.in
> > > > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > > > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > > > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > > > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > > > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > > > prompt "barebox "
> > > > bool
> > > > help
> > > > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > > > Directories and files to add to the builtin barebox
> > > > environment.
> > > > Relative paths must be relative to the platformconfigdir.
> > > >
> > > > +config BAREBOX_REPRODUCIBLE
> > > > + prompt "build reproducibly"
> > > > + bool
> > > > + default n
> > > > + help
> > > > + With this, fake timestamps are injected to the barebox build,
> > > > so the
> > > > + resulting barebox images are identical when the sources
> > > > haven't been
> > > > + changed.
> > > > +
> > > > config BAREBOX_BAREBOXENV
> > > > prompt "install 'bareboxenv'"
> > > > bool
> > > > diff --git a/rules/barebox.make b/rules/barebox.make
> > > > index a6ec3c394..ad160b3ba 100644
> > > > --- a/rules/barebox.make
> > > > +++ b/rules/barebox.make
> > > > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > > > KCONFIG_NOTIMESTAMP=1 \
> > > > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> > > >
> > > > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > > > +BAREBOX_FAKE_TIMESTAMP := \
> > > > + $(shell echo $(BAREBOX_VERSION) | \
> > > > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01
> > > > 00:00:00/")
> > > > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > > > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > > > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > > > + KBUILD_BUILD_VERSION="0" \
> > > > + KBUILD_BUILD_USER="ptxdist" \
> > > > + KBUILD_BUILD_HOST="ptxdist"
> >
> > For the sake of completeness: the _USER and _HOST variables are *not yet*
> > incorporated into the resulting binaries by barebox (at least with my
> > configuration).
>
> Hmm, then something is wrong here. scripts/mkcompile_h has always used
> these variables.
Yes, it generates #defines, but apparently these are not used anywhere in the
barebox source tree, thus they do not end up in the binary.
>
> > > What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> > > hammer that can cause other problems. I'd like to understand what happens
> > > here first.
> >
> > It's an opt-in hammer at least. One annoyance that I observed is, that
> > menuconfig barebox also does build host-faketime and a bunch of other things
> > that are not a dependency of the host-faketime package.
>
> That's the usual early build stuff, right? host-pkg-config and things like
> that.
>
I guess so. It's host-chrpath, host-pkg-config, kernel-header, host-attr, host-
autotools-*, host-acl, host-fakeroot, host-m4 to name a few.
- Florian
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Hi Michael,
Am Dienstag, den 16.10.2018, 12:09 +0200 schrieb Michael Olbrich:
> On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > Inject fake timestamps into the build via faketime to produce the exact
> > same binaries.
>
> I'd like to see this patch split in two.
> One to handle the KBUILD_BUILD_ stuff. That should be done like the kernel
> with the symlink magic, but otherwise unconditionally (for now).
> Use SOURCE_DATE_EPOCH for the timestamp. I want to change the kernel to use
> this too.
SOURCE_DATE_EPOCH uses the ptxdist release number for the timestamp, thus the
timestamp changes with every ptxdist release. I'd prefer to *really* get the
same binary when the source code was not changed at all. A ptxdist version bump
has a high potential of not touching anything barebox related.
I also do not like mangling the barebox version into a timestamp, but I couldn't
come up with any better idea.
>
> Another patch for the rest. I'm not quite sure how I want to handle this.
> My current plan is to have a setup option: Developers can use it to disable
> reproducible builds to see stuff like the build timestamps etc.
> CI and release builds have it enabled by default. I currently don't have
> any reason do disable reproducibility for releases.
>
> What do you think?
My main use case for reproducible builds is RAUC. Since bootloader updates are
especially hairy on some platforms, I'm happy about having the ability to spare
a few of them (while retaining an update mechanism that ensures a persistent
state of the system *and* without using prebuilt bootloaders).
However, I do not mind if ptxdist by default builds more packages reproducibly.
-Florian
>
> Michael
>
> > Signed-off-by: Florian Bäuerle
> > ---
> > platforms/barebox.in | 10 ++
> > rules/barebox.make | 16 +++-
> > 2 files changed, 25 insertions(+), 1 deletion(-)
> >
> > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > index 645366e93..5bc2f08b6 100644
> > --- a/platforms/barebox.in
> > +++ b/platforms/barebox.in
> > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > prompt "barebox "
> > bool
> > help
> > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > Directories and files to add to the builtin barebox environment.
> > Relative paths must be relative to the platformconfigdir.
> >
> > +config BAREBOX_REPRODUCIBLE
> > + prompt "build reproducibly"
> > + bool
> > + default n
> > + help
> > + With this, fake timestamps are injected to the barebox build, so the
> > + resulting barebox images are identical when the sources haven't been
> > + changed.
> > +
> > config BAREBOX_BAREBOXENV
> > prompt "install 'bareboxenv'"
> > bool
> > diff --git a/rules/barebox.make b/rules/barebox.make
> > index a6ec3c394..ad160b3ba 100644
> > --- a/rules/barebox.make
> > +++ b/rules/barebox.make
> > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > KCONFIG_NOTIMESTAMP=1 \
> > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> >
> > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > +BAREBOX_FAKE_TIMESTAMP := \
> > + $(shell echo $(BAREBOX_VERSION) | \
> > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > + KBUILD_BUILD_VERSION="0" \
> > + KBUILD_BUILD_USER="ptxdist" \
> > + KBUILD_BUILD_HOST="ptxdist"
> > +else
> > +BAREBOX_FAKETIME :=
> > +endif
> > +
> > BAREBOX_MAKEVARS := \
> > V=$(PTXDIST_VERBOSE) \
> > HOSTCC=$(HOSTCC) \
> > @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> > endif
> >
> > @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> > - $(MAKE) $(BAREBOX_MAKEVARS)
> > + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> > @$(call touch)
> >
> > # ---
> > -
> > --
> > 2.19.0
> >
> > ___
> > ptxdist mailing list
> > ptxdist@pengutronix.de
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> Inject fake timestamps into the build via faketime to produce the exact
> same binaries.
I'd like to see this patch split in two.
One to handle the KBUILD_BUILD_ stuff. That should be done like the kernel
with the symlink magic, but otherwise unconditionally (for now).
Use SOURCE_DATE_EPOCH for the timestamp. I want to change the kernel to use
this too.
Another patch for the rest. I'm not quite sure how I want to handle this.
My current plan is to have a setup option: Developers can use it to disable
reproducible builds to see stuff like the build timestamps etc.
CI and release builds have it enabled by default. I currently don't have
any reason do disable reproducibility for releases.
What do you think?
Michael
> Signed-off-by: Florian Bäuerle
> ---
> platforms/barebox.in | 10 ++
> rules/barebox.make | 16 +++-
> 2 files changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/platforms/barebox.in b/platforms/barebox.in
> index 645366e93..5bc2f08b6 100644
> --- a/platforms/barebox.in
> +++ b/platforms/barebox.in
> @@ -13,6 +13,7 @@ menuconfig BAREBOX
> select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> prompt "barebox "
> bool
> help
> @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> Directories and files to add to the builtin barebox environment.
> Relative paths must be relative to the platformconfigdir.
>
> +config BAREBOX_REPRODUCIBLE
> + prompt "build reproducibly"
> + bool
> + default n
> + help
> + With this, fake timestamps are injected to the barebox build, so the
> + resulting barebox images are identical when the sources haven't been
> + changed.
> +
> config BAREBOX_BAREBOXENV
> prompt "install 'bareboxenv'"
> bool
> diff --git a/rules/barebox.make b/rules/barebox.make
> index a6ec3c394..ad160b3ba 100644
> --- a/rules/barebox.make
> +++ b/rules/barebox.make
> @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> KCONFIG_NOTIMESTAMP=1 \
> pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
>
> +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> +BAREBOX_FAKE_TIMESTAMP := \
> + $(shell echo $(BAREBOX_VERSION) | \
> + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> +BAREBOX_ENV := $(BAREBOX_ENV) \
> + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> + KBUILD_BUILD_VERSION="0" \
> + KBUILD_BUILD_USER="ptxdist" \
> + KBUILD_BUILD_HOST="ptxdist"
> +else
> +BAREBOX_FAKETIME :=
> +endif
> +
> BAREBOX_MAKEVARS := \
> V=$(PTXDIST_VERBOSE) \
> HOSTCC=$(HOSTCC) \
> @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> endif
>
> @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> - $(MAKE) $(BAREBOX_MAKEVARS)
> + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> @$(call touch)
>
> #
>
> --
> 2.19.0
>
> ___
> ptxdist mailing list
> ptxdist@pengutronix.de
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Fri, Oct 12, 2018 at 12:28:19PM +, Baeuerle, Florian wrote:
> Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> > On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > > Inject fake timestamps into the build via faketime to produce the exact
> > > same binaries.
> > >
> > > Signed-off-by: Florian Bäuerle
> > > ---
> > > platforms/barebox.in | 10 ++
> > > rules/barebox.make | 16 +++-
> > > 2 files changed, 25 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > > index 645366e93..5bc2f08b6 100644
> > > --- a/platforms/barebox.in
> > > +++ b/platforms/barebox.in
> > > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > > prompt "barebox "
> > > bool
> > > help
> > > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > > Directories and files to add to the builtin barebox environment.
> > > Relative paths must be relative to the platformconfigdir.
> > >
> > > +config BAREBOX_REPRODUCIBLE
> > > + prompt "build reproducibly"
> > > + bool
> > > + default n
> > > + help
> > > + With this, fake timestamps are injected to the barebox build, so the
> > > + resulting barebox images are identical when the sources haven't been
> > > + changed.
> > > +
> > > config BAREBOX_BAREBOXENV
> > > prompt "install 'bareboxenv'"
> > > bool
> > > diff --git a/rules/barebox.make b/rules/barebox.make
> > > index a6ec3c394..ad160b3ba 100644
> > > --- a/rules/barebox.make
> > > +++ b/rules/barebox.make
> > > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > > KCONFIG_NOTIMESTAMP=1 \
> > > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> > >
> > > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > > +BAREBOX_FAKE_TIMESTAMP := \
> > > + $(shell echo $(BAREBOX_VERSION) | \
> > > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> > > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > > + KBUILD_BUILD_VERSION="0" \
> > > + KBUILD_BUILD_USER="ptxdist" \
> > > + KBUILD_BUILD_HOST="ptxdist"
>
> For the sake of completeness: the _USER and _HOST variables are *not yet*
> incorporated into the resulting binaries by barebox (at least with my
> configuration).
Hmm, then something is wrong here. scripts/mkcompile_h has always used
these variables.
> >
> > What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> > hammer that can cause other problems. I'd like to understand what happens
> > here first.
>
> It's an opt-in hammer at least. One annoyance that I observed is, that
> menuconfig barebox also does build host-faketime and a bunch of other things
> that are not a dependency of the host-faketime package.
That's the usual early build stuff, right? host-pkg-config and things like
that.
Michael
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > Inject fake timestamps into the build via faketime to produce the exact
> > same binaries.
> >
> > Signed-off-by: Florian Bäuerle
> > ---
> > platforms/barebox.in | 10 ++
> > rules/barebox.make | 16 +++-
> > 2 files changed, 25 insertions(+), 1 deletion(-)
> >
> > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > index 645366e93..5bc2f08b6 100644
> > --- a/platforms/barebox.in
> > +++ b/platforms/barebox.in
> > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > prompt "barebox "
> > bool
> > help
> > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > Directories and files to add to the builtin barebox environment.
> > Relative paths must be relative to the platformconfigdir.
> >
> > +config BAREBOX_REPRODUCIBLE
> > + prompt "build reproducibly"
> > + bool
> > + default n
> > + help
> > + With this, fake timestamps are injected to the barebox build, so the
> > + resulting barebox images are identical when the sources haven't been
> > + changed.
> > +
> > config BAREBOX_BAREBOXENV
> > prompt "install 'bareboxenv'"
> > bool
> > diff --git a/rules/barebox.make b/rules/barebox.make
> > index a6ec3c394..ad160b3ba 100644
> > --- a/rules/barebox.make
> > +++ b/rules/barebox.make
> > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > KCONFIG_NOTIMESTAMP=1 \
> > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> >
> > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > +BAREBOX_FAKE_TIMESTAMP := \
> > + $(shell echo $(BAREBOX_VERSION) | \
> > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > + KBUILD_BUILD_VERSION="0" \
> > + KBUILD_BUILD_USER="ptxdist" \
> > + KBUILD_BUILD_HOST="ptxdist"
For the sake of completeness: the _USER and _HOST variables are *not yet*
incorporated into the resulting binaries by barebox (at least with my
configuration).
>
> What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> hammer that can cause other problems. I'd like to understand what happens
> here first.
It's an opt-in hammer at least. One annoyance that I observed is, that
menuconfig barebox also does build host-faketime and a bunch of other things
that are not a dependency of the host-faketime package.
- Florian
>
> Michael
>
> > +else
> > +BAREBOX_FAKETIME :=
> > +endif
> > +
> > BAREBOX_MAKEVARS := \
> > V=$(PTXDIST_VERBOSE) \
> > HOSTCC=$(HOSTCC) \
> > @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> > endif
> >
> > @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> > - $(MAKE) $(BAREBOX_MAKEVARS)
> > + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> > @$(call touch)
> >
> > # ---
> > -
> > --
> > 2.19.0
> >
> > ___
> > ptxdist mailing list
> > ptxdist@pengutronix.de
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Am Freitag, den 12.10.2018, 14:01 +0200 schrieb Michael Olbrich:
> On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> > Inject fake timestamps into the build via faketime to produce the exact
> > same binaries.
> >
> > Signed-off-by: Florian Bäuerle
> > ---
> > platforms/barebox.in | 10 ++
> > rules/barebox.make | 16 +++-
> > 2 files changed, 25 insertions(+), 1 deletion(-)
> >
> > diff --git a/platforms/barebox.in b/platforms/barebox.in
> > index 645366e93..5bc2f08b6 100644
> > --- a/platforms/barebox.in
> > +++ b/platforms/barebox.in
> > @@ -13,6 +13,7 @@ menuconfig BAREBOX
> > select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> > select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> > select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> > + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> > prompt "barebox "
> > bool
> > help
> > @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> > Directories and files to add to the builtin barebox environment.
> > Relative paths must be relative to the platformconfigdir.
> >
> > +config BAREBOX_REPRODUCIBLE
> > + prompt "build reproducibly"
> > + bool
> > + default n
> > + help
> > + With this, fake timestamps are injected to the barebox build, so the
> > + resulting barebox images are identical when the sources haven't been
> > + changed.
> > +
> > config BAREBOX_BAREBOXENV
> > prompt "install 'bareboxenv'"
> > bool
> > diff --git a/rules/barebox.make b/rules/barebox.make
> > index a6ec3c394..ad160b3ba 100644
> > --- a/rules/barebox.make
> > +++ b/rules/barebox.make
> > @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> > KCONFIG_NOTIMESTAMP=1 \
> > pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
> >
> > +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> > +BAREBOX_FAKE_TIMESTAMP := \
> > + $(shell echo $(BAREBOX_VERSION) | \
> > + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> > +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> > +BAREBOX_ENV := $(BAREBOX_ENV) \
> > + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> > + KBUILD_BUILD_VERSION="0" \
> > + KBUILD_BUILD_USER="ptxdist" \
> > + KBUILD_BUILD_HOST="ptxdist"
>
> What is not covered by the KBUILD_BUILD_* variables? faketime is a big
> hammer that can cause other problems. I'd like to understand what happens
> here first.
According to my observations, only the compression. A few compressors are
capable of not including timestamps (e.g. gzip: --no-name), but others are not
(lzop).
To fix that, there are a few options I do not like:
1. fix the in the barebox build system and implement ugly hacks so this works
for lzop (overwriting an lzo header field) and other compressors that do not
support this out of the box.
2. fix this in ptxdist, probably by doing the same ugly stuff before copying
images to IMAGEDIR
3. fix the problems in lzop and other compressors that do not support faking or
skipping those header fields.
The thirds option is still the best, but it will take ages to reach
distributions.
- Florian
>
> Michael
>
> > +else
> > +BAREBOX_FAKETIME :=
> > +endif
> > +
> > BAREBOX_MAKEVARS := \
> > V=$(PTXDIST_VERBOSE) \
> > HOSTCC=$(HOSTCC) \
> > @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> > endif
> >
> > @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> > - $(MAKE) $(BAREBOX_MAKEVARS)
> > + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> > @$(call touch)
> >
> > # ---
> > -
> > --
> > 2.19.0
> >
> > ___
> > ptxdist mailing list
> > ptxdist@pengutronix.de
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
On Thu, Oct 11, 2018 at 02:19:17PM +, Baeuerle, Florian wrote:
> Inject fake timestamps into the build via faketime to produce the exact
> same binaries.
>
> Signed-off-by: Florian Bäuerle
> ---
> platforms/barebox.in | 10 ++
> rules/barebox.make | 16 +++-
> 2 files changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/platforms/barebox.in b/platforms/barebox.in
> index 645366e93..5bc2f08b6 100644
> --- a/platforms/barebox.in
> +++ b/platforms/barebox.in
> @@ -13,6 +13,7 @@ menuconfig BAREBOX
> select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> prompt "barebox "
> bool
> help
> @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> Directories and files to add to the builtin barebox environment.
> Relative paths must be relative to the platformconfigdir.
>
> +config BAREBOX_REPRODUCIBLE
> + prompt "build reproducibly"
> + bool
> + default n
> + help
> + With this, fake timestamps are injected to the barebox build, so the
> + resulting barebox images are identical when the sources haven't been
> + changed.
> +
> config BAREBOX_BAREBOXENV
> prompt "install 'bareboxenv'"
> bool
> diff --git a/rules/barebox.make b/rules/barebox.make
> index a6ec3c394..ad160b3ba 100644
> --- a/rules/barebox.make
> +++ b/rules/barebox.make
> @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> KCONFIG_NOTIMESTAMP=1 \
> pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
>
> +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> +BAREBOX_FAKE_TIMESTAMP := \
> + $(shell echo $(BAREBOX_VERSION) | \
> + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> +BAREBOX_ENV := $(BAREBOX_ENV) \
> + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> + KBUILD_BUILD_VERSION="0" \
> + KBUILD_BUILD_USER="ptxdist" \
> + KBUILD_BUILD_HOST="ptxdist"
What is not covered by the KBUILD_BUILD_* variables? faketime is a big
hammer that can cause other problems. I'd like to understand what happens
here first.
Michael
> +else
> +BAREBOX_FAKETIME :=
> +endif
> +
> BAREBOX_MAKEVARS := \
> V=$(PTXDIST_VERBOSE) \
> HOSTCC=$(HOSTCC) \
> @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> endif
>
> @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> - $(MAKE) $(BAREBOX_MAKEVARS)
> + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> @$(call touch)
>
> #
>
> --
> 2.19.0
>
> ___
> ptxdist mailing list
> ptxdist@pengutronix.de
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0|
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- |
___
ptxdist mailing list
ptxdist@pengutronix.de
Re: [ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
I tested this for phycore-am335x and phycard-imx27 bootloaders. I get the same
binaries from different machines, but I think anyone using that feature should
first test it before putting any trust into it.
This is a quick implementation that I primarily need for my RAUC bundles (for
not overwriting unchanged bootloaders).
I think this could me implemented as a ptxdist- or platform option that would
theoretically allow building every package reproducibly.
Some packages need special treatment of course, but that could be done with:
ifdef PTXDIST_REPRODUCIBLE
PKG_ENV := $(PKG_ENV) SOME_OTHER_VAR=1
endif
or a new variable:
PKG_REPRODUCIBLE_ENV := ...
Any thoughts or ideas on this?
-Florian
Am Donnerstag, den 11.10.2018, 14:19 + schrieb Baeuerle, Florian:
> Inject fake timestamps into the build via faketime to produce the exact
> same binaries.
>
> Signed-off-by: Florian Bäuerle
> ---
> platforms/barebox.in | 10 ++
> rules/barebox.make | 16 +++-
> 2 files changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/platforms/barebox.in b/platforms/barebox.in
> index 645366e93..5bc2f08b6 100644
> --- a/platforms/barebox.in
> +++ b/platforms/barebox.in
> @@ -13,6 +13,7 @@ menuconfig BAREBOX
> select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
> select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
> select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
> + select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
> prompt "barebox "
> bool
> help
> @@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
> Directories and files to add to the builtin barebox environment.
> Relative paths must be relative to the platformconfigdir.
>
> +config BAREBOX_REPRODUCIBLE
> + prompt "build reproducibly"
> + bool
> + default n
> + help
> + With this, fake timestamps are injected to the barebox build, so the
> + resulting barebox images are identical when the sources haven't been
> + changed.
> +
> config BAREBOX_BAREBOXENV
> prompt "install 'bareboxenv'"
> bool
> diff --git a/rules/barebox.make b/rules/barebox.make
> index a6ec3c394..ad160b3ba 100644
> --- a/rules/barebox.make
> +++ b/rules/barebox.make
> @@ -45,6 +45,20 @@ BAREBOX_ENV := \
> KCONFIG_NOTIMESTAMP=1 \
> pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
>
> +ifdef PTXCONF_BAREBOX_REPRODUCIBLE
> +BAREBOX_FAKE_TIMESTAMP := \
> + $(shell echo $(BAREBOX_VERSION) | \
> + sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
> +BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
> +BAREBOX_ENV := $(BAREBOX_ENV) \
> + KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
> + KBUILD_BUILD_VERSION="0" \
> + KBUILD_BUILD_USER="ptxdist" \
> + KBUILD_BUILD_HOST="ptxdist"
> +else
> +BAREBOX_FAKETIME :=
> +endif
> +
> BAREBOX_MAKEVARS := \
> V=$(PTXDIST_VERBOSE) \
> HOSTCC=$(HOSTCC) \
> @@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
> endif
>
> @+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
> - $(MAKE) $(BAREBOX_MAKEVARS)
> + $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
> @$(call touch)
>
> # -
> ---
> --
> 2.19.0
>
> ___
> ptxdist mailing list
> ptxdist@pengutronix.de
___
ptxdist mailing list
ptxdist@pengutronix.de
[ptxdist] [PATCH 2/3] barebox: add option for building barebox reproducibly
Inject fake timestamps into the build via faketime to produce the exact
same binaries.
Signed-off-by: Florian Bäuerle
---
platforms/barebox.in | 10 ++
rules/barebox.make | 16 +++-
2 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/platforms/barebox.in b/platforms/barebox.in
index 645366e93..5bc2f08b6 100644
--- a/platforms/barebox.in
+++ b/platforms/barebox.in
@@ -13,6 +13,7 @@ menuconfig BAREBOX
select HOST_OPENSSL if BAREBOX_NEEDS_HOST_OPENSSL
select HOST_IMX_CST if BAREBOX_NEEDS_HOST_IMX_CST
select HOST_LZOP if BAREBOX_NEEDS_HOST_LZOP
+ select HOST_FAKETIME if BAREBOX_REPRODUCIBLE
prompt "barebox "
bool
help
@@ -50,6 +51,15 @@ config BAREBOX_EXTRA_ENV_PATH
Directories and files to add to the builtin barebox environment.
Relative paths must be relative to the platformconfigdir.
+config BAREBOX_REPRODUCIBLE
+ prompt "build reproducibly"
+ bool
+ default n
+ help
+ With this, fake timestamps are injected to the barebox build, so the
+ resulting barebox images are identical when the sources haven't been
+ changed.
+
config BAREBOX_BAREBOXENV
prompt "install 'bareboxenv'"
bool
diff --git a/rules/barebox.make b/rules/barebox.make
index a6ec3c394..ad160b3ba 100644
--- a/rules/barebox.make
+++ b/rules/barebox.make
@@ -45,6 +45,20 @@ BAREBOX_ENV := \
KCONFIG_NOTIMESTAMP=1 \
pkg_wrapper_blacklist="$(BAREBOX_WRAPPER_BLACKLIST)"
+ifdef PTXCONF_BAREBOX_REPRODUCIBLE
+BAREBOX_FAKE_TIMESTAMP := \
+ $(shell echo $(BAREBOX_VERSION) | \
+ sed -E "s/([0-9]{4})\.([0-9]{1,2}).*/\1-\2-01 00:00:00/")
+BAREBOX_FAKETIME := faketime -f "$(BAREBOX_FAKE_TIMESTAMP)"
+BAREBOX_ENV := $(BAREBOX_ENV) \
+ KBUILD_BUILD_TIMESTAMP="$(BAREBOX_FAKE_TIMESTAMP)" \
+ KBUILD_BUILD_VERSION="0" \
+ KBUILD_BUILD_USER="ptxdist" \
+ KBUILD_BUILD_HOST="ptxdist"
+else
+BAREBOX_FAKETIME :=
+endif
+
BAREBOX_MAKEVARS := \
V=$(PTXDIST_VERBOSE) \
HOSTCC=$(HOSTCC) \
@@ -113,7 +127,7 @@ ifdef PTXCONF_BAREBOX_EXTRA_ENV
endif
@+cd $(BAREBOX_DIR) && $(BAREBOX_PATH) $(BAREBOX_ENV) \
- $(MAKE) $(BAREBOX_MAKEVARS)
+ $(BAREBOX_FAKETIME) $(MAKE) $(BAREBOX_MAKEVARS)
@$(call touch)
#
--
2.19.0
___
ptxdist mailing list
ptxdist@pengutronix.de
