Bug#769844: linux: please make linux build reproducibly
Earlier today I uploaded version 4.1.1-1~exp1, with some of the changes
here. I built it once on ext4 and then on xfs, hoping to change
directory ordering.
A summary of the debbindiff:
1. timestamps are different on all files in control.tar.gz
2. timestamps are different on many files in data.tar.{gz,xz}
3. anchor IDs are different in generated HTML docs
4. 'Author' and 'Manual' fields in the manual page header and footer differ
where the manual page is generated from headers referenced by multiple
DocBook files
I believe items 1 and 2 are covered by #759886.
I investigated item 3 further and have now fixed it in svn.
Item 4 could be fixed by your generate-manpage-in-deterministic
-order.patch but I would like to find a cleaner solution to it.
Ben.
--
Ben Hutchings
Hoare's Law of Large Problems:
Inside every large problem is a small problem struggling to get out.
signature.asc
Description: This is a digitally signed message part
Bug#769844: linux: please make linux build reproducibly
On Tue, 2015-07-07 at 17:26 +0100, Ben Hutchings wrote:
Earlier today I uploaded version 4.1.1-1~exp1, with some of the changes
here. I built it once on ext4 and then on xfs, hoping to change
directory ordering.
A summary of the debbindiff:
1. timestamps are different on all files in control.tar.gz
2. timestamps are different on many files in data.tar.{gz,xz}
3. anchor IDs are different in generated HTML docs
4. 'Author' and 'Manual' fields in the manual page header and footer differ
where the manual page is generated from headers referenced by multiple
DocBook files
I believe items 1 and 2 are covered by #759886.
I investigated item 3 further and have now fixed it in svn.
Item 4 could be fixed by your generate-manpage-in-deterministic
-order.patch but I would like to find a cleaner solution to it.
I've fixed item 4 in svn by annotating and filtering out duplicates
from the DocBook files.
I also found and fixed an instance of using the current time (for the
date in manual pages).
Ben.
--
Ben Hutchings
If the facts do not conform to your theory, they must be disposed of.
signature.asc
Description: This is a digitally signed message part
Bug#769844: linux: please make linux build reproducibly
On Wed, 2015-01-07 at 20:49 +0100, Jérémy Bobbio wrote:
[...]
With the attached patch, linux can be built reproducibly with the
current experimental framework. The following changes have been
introduced since the previous patch:
* Files in the source tarball are added in a stable order.
* kernel-doc is made deterministic when parsing `@foo()` in
descriptions.
* The last set of manpages when generating manpages from files
referenced in multiple sections of the documentation is
deterministic.
Some `@foo()` constructs in the kernel should probably be fixed to be
just `foo()`. It's unclear what should be used for functions passed as
parameters.
Hardly anyone seems to use the output of kernel-doc so the sources are
of very variable quality. Function pointers aren't functions so I think
they should be @foo.
Moving on to your patch:
Most of the added patches should go upstream and therefore belong under
bugfix/all/ not debian/. Please can you also add a sign-off to these
(after reading what Documentation/SubmittingPatches says about that).
diff -Nru
linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
---
linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
1970-01-01 01:00:00.0 +0100
+++
linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
2015-01-07 16:34:00.0 +0100
@@ -0,0 +1,55 @@
+Description: generate manpages in deterministic order
+ When header files are referenced multiple times in different
+ documentation parts, manpages for the various functions will also
+ be generated multiple times.
[...]
I don't yet understand this, so I'll come back to it later.
diff -Nru
linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
---
linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
1970-01-01 01:00:00.0 +0100
+++
linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
2015-01-07 16:43:31.0 +0100
@@ -0,0 +1,21 @@
+Description: parse kernel-doc deterministically
+ Regular expressions for highlights in kernel-doc are stored in a Perl
+ hash. These hashes are ordered differently for each Perl run. This will
+ prevent kernel-doc to behave deterministically when parsing
+ “@foo()” as in some runs it will be interpreted as a parameter and
+ in the others it will be interpreted as a function.
+ .
+ We now sort the %highlights hash to get the same behavior on every run.
+Author: Jérémy Bobbio lu...@debian.org
+
+--- linux-3.16.7-ckt2.orig/scripts/kernel-doc
linux-3.16.7-ckt2/scripts/kernel-doc
+@@ -2586,7 +2586,7 @@ $kernelversion = get_kernel_version();
+
+ # generate a sequence of code that will splice in highlighting information
+ # using the s// operator.
+-foreach my $pattern (keys %highlights) {
++foreach my $pattern (sort keys %highlights) {
+ # print STDERR scanning pattern:$pattern,
highlight:($highlights{$pattern})\n;
+ $dohighlight .= \$contents =~ s:$pattern:$highlights{$pattern}:gs;\n;
+ }
OK, applied.
diff -Nru
linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
---
linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
1970-01-01 01:00:00.0 +0100
+++
linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
2015-01-05 14:50:58.0 +0100
@@ -0,0 +1,26 @@
+Description: parse Debian KBUILD_BUILD_TIMESTAMP in gen_initramfs_list.sh
+ gen_initramfs_list.sh assumes that KBUILD_BUILD_TIMESTAMP only contains a
+ date. In Debian, it's a longer string looking like:
+ “Debian 3.16.7-2 (2014-11-06)”. Here we adapt the script to parse Debian
+ format.
+ .
+ This change is necessary to have reproducible builds as the date will
+ then be used as mtime for files in initramfs archive instead of the
+ current time.
+Author: Jérémy Bobbio lu...@debian.org
+Bug-Debian: https://bugs.debian.org/769844
+Forwarded: not-needed
+
+--- linux-3.16.7-ckt2.orig/scripts/gen_initramfs_list.sh
linux-3.16.7-ckt2/scripts/gen_initramfs_list.sh
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+- timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP +%s ||
:)
++ source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++ sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++ timestamp=$(date -d$source_date +%s || :)
+
Bug#769844: linux: please make linux build reproducibly
On Thu, 2015-01-08 at 09:16 +, Ian Campbell wrote:
On Wed, 2015-01-07 at 20:49 +0100, Jérémy Bobbio wrote:
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+- timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP
+%s || :)
++ source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++ sed -e
's/.*(\([0-9-]\+\)).*/\1/')
++ timestamp=$(date -d$source_date +%s || :)
This solution may not work. The patched source can be built with a
normal timestamp override, via linux-source.
The above construction will work given a standard date in
KBUILD_BUILD_TIMESTAMP. The sed expression only match parenthesises.
I am open to other suggestions.
How about changing debian/rules.real to set some other variable (e.g.
DEB_BUILD_TIMESTAMP, or whatever) to our special version string and
patching the relevant code (not here, but the scripts/mkcompile_h one)
to prefer $DEB_BUILD_TIMESTAMP but still support $KBUILD_BUILD_TIMESTAMP
as a fallback?
I think we would still need to set KBUILD_BUILD_TIMESTAMP to the
debian/changelog date in rules.real.
I've implemented something pretty similar to that:
- $KBUILD_BUILD_TIMESTAMP is set to the date in the changelog
- $KBUILD_BUILD_VERSION_TIMESTAMP is set to what we want in
utsname::version
- mkcompile_h prefers $KBUILD_BUILD_VERSION_TIMESTAMP over
$KBUILD_BUILD_TIMESTAMP
Ben.
--
Ben Hutchings
For every action, there is an equal and opposite criticism. - Harrison
signature.asc
Description: This is a digitally signed message part
Bug#769844: linux: please make linux build reproducibly
On Wed, 2015-01-07 at 20:49 +0100, Jérémy Bobbio wrote:
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+-timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP
+%s || :)
++source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++sed -e
's/.*(\([0-9-]\+\)).*/\1/')
++timestamp=$(date -d$source_date +%s || :)
This solution may not work. The patched source can be built with a
normal timestamp override, via linux-source.
The above construction will work given a standard date in
KBUILD_BUILD_TIMESTAMP. The sed expression only match parenthesises.
I am open to other suggestions.
How about changing debian/rules.real to set some other variable (e.g.
DEB_BUILD_TIMESTAMP, or whatever) to our special version string and
patching the relevant code (not here, but the scripts/mkcompile_h one)
to prefer $DEB_BUILD_TIMESTAMP but still support $KBUILD_BUILD_TIMESTAMP
as a fallback?
I think we would still need to set KBUILD_BUILD_TIMESTAMP to the
debian/changelog date in rules.real.
Ian.
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#769844: linux: please make linux build reproducibly
Bastian Blank:
On Mon, Jan 05, 2015 at 06:56:10PM +0100, Jérémy Bobbio wrote:
* linux-source: mtimes of many files differ. Would it be ok to just
create the tarball with a single timestamp (`tar --mtime=`)?
I've used find+touch instead. See attached patch.
Why?
Good question. It's the way we are doing it in dh_builddeb. But for
kernel-source, using `--mtime` is more straight forward. This is done in
the attached patch.
I also stumbled on a variation of the Installed-Size field in one of the
.deb. But this is a different topic.
Is this value computed in a stable way?
The value can differ greatly depending on the underlying filesystem.
This is to be addressed at dpkg's level. josch is working on it.
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+- timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP +%s || :)
++ source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++ sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++ timestamp=$(date -d$source_date +%s || :)
This solution may not work. The patched source can be built with a
normal timestamp override, via linux-source.
The above construction will work given a standard date in
KBUILD_BUILD_TIMESTAMP. The sed expression only match parenthesises.
I am open to other suggestions.
@@ -87,6 +87,8 @@
rm -rf '$@' '$(DIR)'
$(call copy_source,$(DIR))
chmod -R u+rw,go=rX '$(DIR)'
+ find '$(DIR)' -depth -newermt '$(SOURCE_DATE)' -print0 | \
+ xargs -0r touch --no-dereference --date='$(SOURCE_DATE)'
-newermt? This does not look really stable.
I don't understand what you mean by that. It's been proven working for
a while in our experiments.
With the attached patch, linux can be built reproducibly with the
current experimental framework. The following changes have been
introduced since the previous patch:
* Files in the source tarball are added in a stable order.
* kernel-doc is made deterministic when parsing `@foo()` in
descriptions.
* The last set of manpages when generating manpages from files
referenced in multiple sections of the documentation is
deterministic.
Some `@foo()` constructs in the kernel should probably be fixed to be
just `foo()`. It's unclear what should be used for functions passed as
parameters.
--
Lunar.''`.
lu...@debian.org: :Ⓐ : # apt-get install anarchism
`. `'`
`-
diff -Nru linux-3.16.7-ckt2/debian/changelog linux-3.16.7-ckt2/debian/changelog
--- linux-3.16.7-ckt2/debian/changelog 2014-12-08 21:03:19.0 +0100
+++ linux-3.16.7-ckt2/debian/changelog 2015-01-07 16:50:40.0 +0100
@@ -1,3 +1,16 @@
+linux (3.16.7-ckt2-1.0~reproducible1) UNRELEASED; urgency=low
+
+ * Make build reproducible:
+- Add a patch to make gen_initramfs.sh parse Debian's KBUILD_BUILD_TIMESTAMP.
+- Add a patch to make kernel-doc parse highlights determistically.
+- Add a patch to force the order when generating manpages from files
+ referenced in multiple sections of the documentation.
+- Use debian/changelog date as mtimes in the generated source tarball.
+- Write files in the source tarball in a stable order.
+- Stop recording current time in gzip headers when compressing documentation.
+
+ -- Jérémy Bobbio lu...@debian.org Wed, 07 Jan 2015 16:50:39 +0100
+
linux (3.16.7-ckt2-1) unstable; urgency=high
* New upstream stable update:
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
--- linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch 1970-01-01 01:00:00.0 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch 2015-01-07 16:34:00.0 +0100
@@ -0,0 +1,55 @@
+Description: generate manpages in deterministic order
+ When header files are referenced multiple times in different
+ documentation parts, manpages for the various functions will also
+ be generated multiple times.
+ .
+ In order to support reproducible builds when multiple jobs are
+ enabled, we need to make the order deterministic. To do so, we
+ scan the documentation, identify sections having similar directives,
+ and generate dependencies making the first section in alphabetical
+ order build last.
+ .
+ With the current kernel, the following dependencies are generated:
+ .
+ Documentation/DocBook/alsa-driver-api.tmpl: Documentation/DocBook/device-drivers.tmpl
+ Documentation/DocBook/gadget.tmpl: Documentation/DocBook/usb.tmpl
+ Documentation/DocBook/genericirq.tmpl: Documentation/DocBook/kernel-api.tmpl
+Author: Jérémy Bobbio
Bug#769844: linux: please make linux build reproducibly
On Mon, Jan 05, 2015 at 06:56:10PM +0100, Jérémy Bobbio wrote:
We are currently experimenting with fixing mtimes in `dh_builddeb`
instead of requiring a new helper. I have also done my latest
experiments without `dh_strip_nondeterminism`. The attached patch adds
the `-n` flag to gzip to compensate.
Okay.
That's how the experimental toolchain now works: the .buildinfo is
generated by dpkg-genbuildinfo, called by dpkg-buildpackage.
Thanks
* linux-source: mtimes of many files differ. Would it be ok to just
create the tarball with a single timestamp (`tar --mtime=`)?
I've used find+touch instead. See attached patch.
Why?
I also stumbled on a variation of the Installed-Size field in one of the
.deb. But this is a different topic.
Is this value computed in a stable way?
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+-timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP +%s || :)
++source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++timestamp=$(date -d$source_date +%s || :)
This solution may not work. The patched source can be built with a
normal timestamp override, via linux-source.
@@ -87,6 +87,8 @@
rm -rf '$@' '$(DIR)'
$(call copy_source,$(DIR))
chmod -R u+rw,go=rX '$(DIR)'
+ find '$(DIR)' -depth -newermt '$(SOURCE_DATE)' -print0 | \
+ xargs -0r touch --no-dereference --date='$(SOURCE_DATE)'
-newermt? This does not look really stable.
Bastian
--
Every living thing wants to survive.
-- Spock, The Ultimate Computer, stardate 4731.3
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#769844: linux: please make linux build reproducibly
Control: unblock -1 by 759886
Jérémy Bobbio:
Bastian Blank:
On Mon, Nov 17, 2014 at 12:46:45AM +0100, Jérémy Bobbio wrote:
The first patch adds call to `dh_strip_nondeterminism` and
`dh_fixmtimes`, both being part of the custom toolchain currently used
for reproducible builds. Hence not tagging the bug with “patch” until
they are integrated in debhelper.
Why does this need new tool instead of being integrated into the
existing ones?
I am not sure which ones you specifically have in mind, but the whole
project is still at the experimental stage. We try to work in
unintrusive ways.
We are currently experimenting with fixing mtimes in `dh_builddeb`
instead of requiring a new helper. I have also done my latest
experiments without `dh_strip_nondeterminism`. The attached patch adds
the `-n` flag to gzip to compensate.
The second patch changes the value of KBUILD_BUILD_TIMESTAMP to a
timestamp parseable by `date`.
Well, no. The string is this way for a reason.
Would a patch against `scripts/gen_initramfs_list.sh` to make it parse
Debian's KBUILD_BUILD_TIMESTAMP be acceptable then? Any other
suggestions?
Implemented in the attached patch.
An unclear aspect is where to add a call to `dh_genbuildinfo` which
generates the .buildinfo [2]. It should be called after all binary
packages have been created.
Not possible, dh_* acts on single binary packages.
Mh… I'm not sure we had realized that. It makes a case to move the
generation of the .buildinfo closer to dpkg-genchanges.
That's how the experimental toolchain now works: the .buildinfo is
generated by dpkg-genbuildinfo, called by dpkg-buildpackage.
* linux-source: mtimes of many files differ. Would it be ok to just
create the tarball with a single timestamp (`tar --mtime=`)?
Looks like a way.
Good. :) I will experiment with this approach and probably add another
patch to this bug report.
I've used find+touch instead. See attached patch.
With the attached patch, my latest build+rebuild showed similar
differences in linux-doc and linux-manual. It is probably related to the
way API documentation is currently extracted from the source code.
I also stumbled on a variation of the Installed-Size field in one of the
.deb. But this is a different topic.
--
Lunar.''`.
lu...@debian.org: :Ⓐ : # apt-get install anarchism
`. `'`
`-
diff -Nru linux-3.16.7-ckt2/debian/changelog linux-3.16.7-ckt2/debian/changelog
--- linux-3.16.7-ckt2/debian/changelog 2014-12-08 21:03:19.0 +0100
+++ linux-3.16.7-ckt2/debian/changelog 2015-01-05 18:24:20.0 +0100
@@ -1,3 +1,11 @@
+linux (3.16.7-ckt2-1.0~reproducible1) UNRELEASED; urgency=low
+
+ * Add a patch to make gen_initramfs.sh parse Debian's KBUILD_BUILD_TIMESTAMP.
+ * Adjust file mtimes before creating source tarball.
+ * Stop recording current time in gzip headers when compressing documentation.
+
+ -- Jérémy Bobbio lu...@debian.org Mon, 05 Jan 2015 14:55:50 +0100
+
linux (3.16.7-ckt2-1) unstable; urgency=high
* New upstream stable update:
--- linux-3.16.7-ckt2/debian/patches/debianparse-debian-build-timestamp.patch 1970-01-01 01:00:00.0 +0100
+++ linux-3.16.7-ckt2/debian/patches/debianparse-debian-build-timestamp.patch 2015-01-05 14:50:58.0 +0100
@@ -0,0 +1,26 @@
+Description: parse Debian KBUILD_BUILD_TIMESTAMP in gen_initramfs_list.sh
+ gen_initramfs_list.sh assumes that KBUILD_BUILD_TIMESTAMP only contains a
+ date. In Debian, it's a longer string looking like:
+ “Debian 3.16.7-2 (2014-11-06)”. Here we adapt the script to parse Debian
+ format.
+ .
+ This change is necessary to have reproducible builds as the date will
+ then be used as mtime for files in initramfs archive instead of the
+ current time.
+Author: Jérémy Bobbio lu...@debian.org
+Bug-Debian: https://bugs.debian.org/769844
+Forwarded: not-needed
+
+--- linux-3.16.7-ckt2.orig/scripts/gen_initramfs_list.sh
linux-3.16.7-ckt2/scripts/gen_initramfs_list.sh
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n $KBUILD_BUILD_TIMESTAMP; then
+- timestamp=$(date -d$KBUILD_BUILD_TIMESTAMP +%s || :)
++ source_date=$(echo $KBUILD_BUILD_TIMESTAMP |
++ sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++ timestamp=$(date -d$source_date +%s || :)
+ if test -n $timestamp; then
+ timestamp=-t $timestamp
+ fi
diff -Nru linux-3.16.7-ckt2/debian/patches/series linux-3.16.7-ckt2/debian/patches/series
--- linux-3.16.7-ckt2/debian/patches/series 2014-12-08 21:00:20.0 +0100
+++ linux-3.16.7-ckt2/debian/patches/series 2015-01-05 14:44:13.0 +0100
@@ -484,3 +484,4 @@
debian/iovec-fix-abi-change-in-3.16.7-ckt1.patch
debian/truncate-fix-abi-change-in-3.16.7-ckt1.patch
debian/perf-fix-abi-change-in-3.16.7-ckt2.patch
Bug#769844: linux: please make linux build reproducibly
Bastian Blank: On Mon, Nov 17, 2014 at 12:46:45AM +0100, Jérémy Bobbio wrote: The first patch adds call to `dh_strip_nondeterminism` and `dh_fixmtimes`, both being part of the custom toolchain currently used for reproducible builds. Hence not tagging the bug with “patch” until they are integrated in debhelper. Why does this need new tool instead of being integrated into the existing ones? I am not sure which ones you specifically have in mind, but the whole project is still at the experimental stage. We try to work in unintrusive ways. The second patch changes the value of KBUILD_BUILD_TIMESTAMP to a timestamp parseable by `date`. Well, no. The string is this way for a reason. Would a patch against `scripts/gen_initramfs_list.sh` to make it parse Debian's KBUILD_BUILD_TIMESTAMP be acceptable then? Any other suggestions? An unclear aspect is where to add a call to `dh_genbuildinfo` which generates the .buildinfo [2]. It should be called after all binary packages have been created. Not possible, dh_* acts on single binary packages. Mh… I'm not sure we had realized that. It makes a case to move the generation of the .buildinfo closer to dpkg-genchanges. * linux-source: mtimes of many files differ. Would it be ok to just create the tarball with a single timestamp (`tar --mtime=`)? Looks like a way. Good. :) I will experiment with this approach and probably add another patch to this bug report. -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- signature.asc Description: Digital signature
Bug#769844: linux: please make linux build reproducibly
Source: linux Version: 3.16.7-2 Severity: wishlist User: reproducible-bui...@lists.alioth.debian.org Usertags: timestamps randomness Control: block -1 by 759886 Hi! I have been doing some experimentation on making linux build reproducibly [1]. With the attached patches, we are down to three binary packages with differences on amd64. The kernel itself and its module can be built reproducibly with our current framework. The first patch adds call to `dh_strip_nondeterminism` and `dh_fixmtimes`, both being part of the custom toolchain currently used for reproducible builds. Hence not tagging the bug with “patch” until they are integrated in debhelper. The second patch changes the value of KBUILD_BUILD_TIMESTAMP to a timestamp parseable by `date`. Otherwise, a timestamp of the current time gets stored in usr/initramfs_data.cpio.gz because `scripts/gen_initramfs_list.sh` will not pass the value of KBUILD_BUILD_TIMESTAMP to `usr/gen_init_cpio`. http://sources.debian.net/src/linux/3.16.7-2/scripts/gen_initramfs_list.sh/?hl302:308#L302 Another solution would be to patch `scripts/gen_initramfs_list.sh` to parse the Debian format of KBUILD_BUILD_TIMESTAMP. An unclear aspect is where to add a call to `dh_genbuildinfo` which generates the .buildinfo [2]. It should be called after all binary packages have been created. For the remaining differences: * linux-doc: many variations due to ids generated in HTML documentation by XSLT processor. This needs to be addressed at that level. * linux-manual: see attached debbindiff output. I don't have good ideas. * linux-source: mtimes of many files differ. Would it be ok to just create the tarball with a single timestamp (`tar --mtime=`)? [1]: https://wiki.debian.org/ReproducibleBuilds [2]: https://wiki.debian.org/ReproducibleBuilds/BuildinfoSpecification -- Lunar.''`. lu...@debian.org: :Ⓐ : # apt-get install anarchism `. `'` `- --- linux-3.16.7/debian/rules.real 2014-11-04 04:41:34.0 + +++ b1/linux-3.16.7/debian/rules.real 2014-11-14 22:07:20.272662604 + @@ -173,12 +173,14 @@ dh_installdocs $(INSTALLDOCS_ARGS) dh_installchangelogs dh_strip + dh_strip_nondeterminism dh_compress dh_fixperms dh_installdeb dh_gencontrol -- $(GENCONTROL_ARGS) dh_md5sums + dh_fixmtimes dh_builddeb -- -Zxz $(BUILDDEB_ARGS) install-dummy: dh_testdir @@ -427,8 +430,10 @@ dh_prep kernel-wedge install-files $(ABINAME) kernel-wedge check $(PACKAGE_NAMES) + dh_strip_nondeterminism dh_fixperms dh_gencontrol + dh_fixmtimes dh_builddeb install-source: PACKAGE_NAME = linux-source-$(VERSION) --- linux-3.16.7/debian/rules.real 2014-11-04 04:41:34.0 + +++ b1/linux-3.16.7/debian/rules.real 2014-11-14 22:07:20.272662604 + @@ -39,7 +39,7 @@ stamp = [ -d $(dir $@) ] || mkdir $(dir $@); touch $@ setup_env := env -u ABINAME -u ARCH -u FEATURESET -u FLAVOUR -u VERSION -u LOCALVERSION -setup_env += DISTRIBUTION_OFFICIAL_BUILD=1 DISTRIBUTOR=$(DISTRIBUTOR) DISTRIBUTION_VERSION=$(SOURCEVERSION) KBUILD_BUILD_TIMESTAMP=$(DISTRIBUTOR) $(SOURCEVERSION) ($(SOURCE_DATE_UTC_ISO)) KBUILD_BUILD_USER=$(word 1,$(subst @, ,$(MAINTAINER))) KBUILD_BUILD_HOST=$(word 2,$(subst @, ,$(MAINTAINER))) +setup_env += DISTRIBUTION_OFFICIAL_BUILD=1 DISTRIBUTOR=$(DISTRIBUTOR) DISTRIBUTION_VERSION=$(SOURCEVERSION) KBUILD_BUILD_TIMESTAMP=$(SOURCE_DATE_UTC_ISO) KBUILD_BUILD_USER=$(word 1,$(subst @, ,$(MAINTAINER))) KBUILD_BUILD_HOST=$(word 2,$(subst @, ,$(MAINTAINER))) MAKE_CLEAN = $(setup_env) $(MAKE) MAKE_SELF := $(MAKE) -f debian/rules.real $(MAKEOVERRIDES) Title: /usr/bin/debbindiff --debug --html debbindiff-manual.html b1/linux-manual-3.16_3.16.7-2.0~reproducible1_all.deb b2/linux-manual-3.16_3.16.7-2.0~reproducible1_all.deb linux-manual-3.16_3.16.7-2.0~reproducible1_all.deb control.tar.gz control.tar md5sums Files in package differs data.tar.xz data.tar kthread_create_on_node.9.gz kthread_create_on_node.9 1 +-- 63 lines: '\ t- 64 This helper function creates and names a kernel thread\. The thread will be stopped: use 65 \fBwake_up_process\fR 66 to start it\. See also 67 \fBkthread_run\fR\. 68 .PP 69 If thread is going to be bound on a particular cpu, give its node in 70 \fInode\fR, to get NUMA affinity for kthread stack, or else give \-1\. When woken, the thread will run 71 \fIthreadfn\fR() with 72 \fIdata\fR 73 as its argument\. 74 \fIthreadfn\fR() can either call 75 \fBdo_exit\fR 76 directly if it is a standalone thread for which no one will call 77 \fBkthread_stop\fR, or return when \*(Aq\fBkthread_should_stop\fR\*(Aq is true (which means 78 \fBkthread_stop\fR 79 has been
Bug#769844: linux: please make linux build reproducibly
On Mon, Nov 17, 2014 at 12:46:45AM +0100, Jérémy Bobbio wrote: The first patch adds call to `dh_strip_nondeterminism` and `dh_fixmtimes`, both being part of the custom toolchain currently used for reproducible builds. Hence not tagging the bug with “patch” until they are integrated in debhelper. Why does this need new tool instead of being integrated into the existing ones? The second patch changes the value of KBUILD_BUILD_TIMESTAMP to a timestamp parseable by `date`. Well, no. The string is this way for a reason. An unclear aspect is where to add a call to `dh_genbuildinfo` which generates the .buildinfo [2]. It should be called after all binary packages have been created. Not possible, dh_* acts on single binary packages. * linux-source: mtimes of many files differ. Would it be ok to just create the tarball with a single timestamp (`tar --mtime=`)? Looks like a way. Bastian -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
