Bug#862059: sbuild: please sign buildinfo files
Package: sbuild
Version: 0.73.0-4
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: infrastructure
Hello,
dpkg-buildpackage typically generates a .changes and .buildinfo file,
and signs both (since at least dpkg 1.18.19).
But when using sbuild, dpkg-buildpackage inside of the build chroot does
not do the signing, but rather sbuild signs the .changes file afterward.
Please could that code be updated to also sign the .buildinfo (if one
was created).
I have not tested the attached patch (yet?) but it explains the issue at
least. Here is typical output where only the .changes file gets signed:
>
> Finished at 20170314-2338
> Build needed 00:00:43, 5660k disc space
> Signature with key 'F2F4A5FC' requested:
> signfile /home/buildd/build/hello_2.10-1+b1_amd64.changes F2F4A5FC
>
> Successfully signed changes file
The relevance/importance of this is that official Debian package builds
produce .buildinfo files now, and dak archives them, but they are not
being signed yet.
Thanks!
-- System Information:
Debian Release: stretch/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: kfreebsd-amd64 (x86_64)
Kernel: kFreeBSD 10.1-0-amd64
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
diff --git a/lib/Sbuild/Build.pm b/lib/Sbuild/Build.pm
index 9a767a2..b196812 100644
--- a/lib/Sbuild/Build.pm
+++ b/lib/Sbuild/Build.pm
@@ -2668,6 +2668,21 @@ sub get_changes {
return $changes;
}
+sub get_buildinfo {
+my $self=shift;
+my $buildinfo;
+
+if ($self->get_conf('BUILD_ARCH_ANY')) {
+ $buildinfo = $self->get('Package_SVersion') . '_' . $self->get('Host Arch') . '.buildinfo';
+} elsif ($self->get_conf('BUILD_ARCH_ALL')) {
+ $buildinfo = $self->get('Package_SVersion') . "_all.buildinfo";
+} elsif ($self->get_conf('BUILD_SOURCE')) {
+ $buildinfo = $self->get('Package_SVersion') . "_source.buildinfo";
+}
+
+return $buildinfo;
+}
+
sub check_space {
my $self = shift;
my @files = @_;
@@ -3169,6 +3184,7 @@ sub close_build_log {
my $key_id = $self->get_conf('KEY_ID');
my $build_dir = $self->get_conf('BUILD_DIR');
my $changes;
+ my $buildinfo;
$self->log(sprintf("Signature with key '%s' requested:\n", $key_id));
$changes = $self->get_changes();
if (!defined($changes)) {
@@ -3176,6 +3192,10 @@ sub close_build_log {
} else {
system('debsign', '--re-sign', "-k$key_id", '--', "$build_dir/$changes");
}
+ $buildinfo = $self->get_buildinfo();
+ if (defined($buildinfo) && stat($buildinfo)) {
+ system('debsign', '--re-sign', "-k$key_id", '--', "$build_dir/$buildinfo");
+ }
if ($self->get_conf('SOURCE_ONLY_CHANGES')) {
my $so_changes = $build_dir . '/' . $self->get('Package_SVersion') . "_source.changes";
if (-r $so_changes) {
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#816439: linux-grsec: feeding the seed with SOURCE_DATE_EPOCH
tags 816439 + patch thanks Hi, > > Contrary to: https://bugs.debian.org/816439, RANDSTRUCT is > > actually compatible with reproducible builds, just need to > > keep randomize_layout_seed.h. > > https://twitter.com/grsecurity/status/704869584218685440 Holger Levsen wrote: > seens like the it should be seeded with > https://reproducible-builds.org/specs/source-date-epoch/ Patch attached! (please read it for the long description) I'm afraid I couldn't find where to do this in linux-grsec Git. So I've attached it in debdiff form instead. There might still be other reproducibility issues after this. Thanks, Regards, -- Steven Chamberlain ste...@pyro.eu.org diff -Nru linux-grsec-4.6.3/debian/patches/features/all/grsec/reproducible-randstruct.patch linux-grsec-4.6.3/debian/patches/features/all/grsec/reproducible-randstruct.patch --- linux-grsec-4.6.3/debian/patches/features/all/grsec/reproducible-randstruct.patch 1970-01-01 00:00:00.0 + +++ linux-grsec-4.6.3/debian/patches/features/all/grsec/reproducible-randstruct.patch 2016-07-12 22:02:33.0 + @@ -0,0 +1,29 @@ +Date: Tue, 12 Jul 2016 22:59:00 +0100 +From: Steven Chamberlain <stev...@debian.org> +Subject: make reproducible randomize_layout_seed.h + +If the SOURCE_DATE_EPOCH environment variable is defined, use the +SHA-256 hash of that value, instead of /dev/urandom, to initialise +randstruct_seed in randomize_layout_seed.h + +This allows the build to be reproducible. In order to force a new seed, +modify the SOURCE_DATE_EPOCH. (In Debian, bumping the package version +and adding a new debian/changelog entry will have that effect). + +Signed-off-by: Steven Chamberlain <stev...@debian.org> + +--- a/scripts/gcc-plugins/gen-random-seed.sh b/scripts/gcc-plugins/gen-random-seed.sh +@@ -1,7 +1,11 @@ + #!/bin/sh + + if [ ! -f "$1" ]; then +- SEED=`od -A n -t x8 -N 32 /dev/urandom | tr -d ' \n'` ++ if [ ! -z "$SOURCE_DATE_EPOCH" ]; then ++ SEED=`echo -n "$SOURCE_DATE_EPOCH" | sha256sum | cut -d" " -f1` ++ else ++ SEED=`od -A n -t x8 -N 32 /dev/urandom | tr -d ' \n'` ++ fi + echo "const char *randstruct_seed = \"$SEED\";" > "$1" + HASH=`echo -n "$SEED" | sha256sum | cut -d" " -f1 | tr -d ' \n'` + echo "#define RANDSTRUCT_HASHED_SEED \"$HASH\"" > "$2" diff -Nru linux-grsec-4.6.3/debian/patches/series-grsec linux-grsec-4.6.3/debian/patches/series-grsec --- linux-grsec-4.6.3/debian/patches/series-grsec 2016-07-07 05:17:23.0 + +++ linux-grsec-4.6.3/debian/patches/series-grsec 2016-07-12 21:57:58.0 + @@ -1 +1,2 @@ features/all/grsec/grsecurity-3.1-4.6.3-201607062159+debian.patch +features/all/grsec/reproducible-randstruct.patch signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#816072: tar: Please upstream --clamp-mtime option
tags 816072 + fixed-upstream thanks tar 1.29 is released today with this feature \o/ Please consider packaging it so that dpkg can fix #759886 Thanks, Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Performance of armhf boards
Hi!
I was wondering what is the performance of various armhf boards, for
package building. Of course, the reproducible-builds team have a lot of
stats already. Below I'm sharing the query I used and the results in
case anyone else is interested in this.
Using https://tests.reproducible-builds.org/reproducible.db
(warning: >300MiB)
$ DATE=$(date -u -d "1 day ago" '+%Y-%m-%d')
$ TIMESPAN_RAW=21
$ sqlite3 -column -header reproducible.db \
"SELECT r.node1 AS buildd, COUNT(r.id) AS pkgs_built,
CAST(AVG(r.build_duration) AS INTEGER) AS avg_duration FROM results AS r WHERE
r.build_duration!='' AND r.build_duration!='0' AND r.build_date >
datetime('$DATE', '-$TIMESPAN_RAW days') GROUP BY r.node1 ORDER BY pkgs_built
DESC"
Figures will vary based on which packages were assigned to which
node, as some are easier to build than others, but I hope over 21
days that variance is smoothed out.
Assuming the nodes had no downtime, we can compare pkgs_built over
the 21-day period to assess performance.
Also avg_duration is meaningful, but will increase where the
reproducible-builds network scheduled more concurrent build jobs on a
node. (Low avg_duration does not always mean high package throughput,
it may just be doing fewer jobs in parallel.)
Finally, the nodes' performance will depend on other factors such as
storage device used, kernel, etc.
Rows are annotated with number of cores, amount of RAM, and board.
builddpkgs_built avg_duration
--
profitbricks-build5-amd64.debian.net 17415 514 # 18x,48G
profitbricks-build1-amd64.debian.net 16720 531 # 17x,48G
profitbricks-build6-i386.debian.net 15348 727 # 18x,48G
profitbricks-build2-i386.debian.net 15214 739 # 17x,48G
wbq0-armhf-rb.debian.net 21702359# 4x,2G;
Wandboard-Quad?
cbxi4b-armhf-rb.debian.net20772582# 4x,4G;
CuBox-i4x4
odxu4-armhf-rb.debian.net 20072255# 8x,2G;
Odroid-XU4 (USB3 SATA SSD)
cbxi4a-armhf-rb.debian.net19962365# 4x,4G;
Cubox-i4x4
cbxi4pro0-armhf-rb.debian.net 19732743# 4x,2G;
CuBox-i4Pro
opi2a-armhf-rb.debian.net 17672922# 4x,2G; OrangePi
Plus2 (USB2 SATA SSD)
odxu4c-armhf-rb.debian.net17422180# 8x,2G;
Odroid-XU4
odxu4b-armhf-rb.debian.net16272295# 8x,2G;
Odroid-XU4
ff2b-armhf-rb.debian.net 15292745# 4x,2G;
Firefly-RK3288 (USB2 SATA SSD)
opi2b-armhf-rb.debian.net 14602738# 4x,2G; OrangePi
Plus2 (USB2 SATA SSD)
ff2a-armhf-rb.debian.net 14352570# 4x,2G;
Firefly-RK3288 (USB3 SATA SSD)
bbx15-armhf-rb.debian.net 11511827# 2x,2G;
BeagleBoard X15 - cool!
rpi2c-armhf-rb.debian.net 11371986# 4x,1G;
Raspberry PI 2
hb0-armhf-rb.debian.net 11342143# 2x,1G;
HummingBoard Pro i2?
bpi0-armhf-rb.debian.net 773 3433# 2x,1G; Banana
Pi?
ff4a-armhf-rb.debian.net 630 1728# 4x,4G;
Firefly-RK3288
rpi2b-armhf-rb.debian.net 626 1972# 4x,1G;
Raspberry PI 2 Model B
wbd0-armhf-rb.debian.net 403 3176# 2x,1G;
Wandboard-Dual (USB2 SATA HDD)
I don't know whether to believe these figures yet!
* wbq0 is impossibly fast for just 4x1GHz cores, 2GB RAM...
* odxu4 looks slightly faster than the other two.
* cbxi4a/b seem no faster than cbxi4pro0 despite twice the RAM?
* ff2a/b show USB3 SSD to be no faster than USB2?
* bbx15 may be able to handle more build jobs (low avg_duration).
* bpi0 may be overloaded (high avg_duration).
* ff4a maybe had downtime, and seems to be under-utilised.
* rpi2b maybe had downtime, or has a slower disk than rpi2c.
* wbd0 slowness is likely due to the magnetic hard drive.
Corrections/suggestions are welcome.
Many thanks to Vagrant for hosting all these armhf nodes!
Regards,
--
Steven Chamberlain
ste...@pyro.eu.org
signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] arm64 reproducible build network
Hi, I'm curious if anyone has tried using a network filesystem in this kind of setup. I would think, "diskless" boards sharing a NAS allows for easier provisioning and probably cheaper storage by centralising it. Though I don't know how that performs in practice? Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Raspi 3 suitable for arm64?
Hi, Karsten Merker wrote: > For the Odroid-C2 there is AFAICS no mainline support at all. There was a pull recently for the SoC used on that board (Amlogic S905): https://lkml.org/lkml/2016/3/1/1401 Does that make mainline support for the Odroid C2 any more likely? I guess substantial work is still needed yet. (Annoying that there are so many cheap boards that claim to be/do so much and yet, are of little practical use if they can only boot a vendor-supplied kernel). Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: : Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Jérémy Bobbio wrote: > [...] It missed another bit. Thanks for double-checking, I hadn't > tested the other change properly. And thanks for fixing this! The changes from diffoscope/48 to 49 have made it 26x faster for this particular test case. Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#813052: Bug#813052: diffoscope takes more than an hour on foreign arch libc6
Jérémy Bobbio wrote:
> Steven Chamberlain:
> > But it will still stat() everything in the containing directory,
> > looking for .debs. It also opens some files and reads them - even
> > decompressing random .gz files along the way!
>
> Are you sure that it is actually decompressing files and not just
> identifying them?
Ah okay, it reads just one block to check its file magic, I think.
> Anyway, I've just pushed another patch to filter by filenames before
> looking at content. This should further improve the situation.
I don't think it worked? It's still doing as before, looking at
text, gzip files and validating the sha1sums in a .buildinfo:
| DEBUG Looking for a dbgsym package for Build Id
4bfc8175c9c53156a7e20d0216bc9fff0d25ae2a (debuglink:
fc8175c9c53156a7e20d0216bc9fff0d25ae2a.debug)
| DEBUG Using TextFile for a/build/.bash_logout
| DEBUG Using TextFile for a/build/.bashrc
| DEBUG Using TextFile for a/build/.profile
| DEBUG Using DebFile for a/build/cpp-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/cpp-4.9_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9-multilib_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-base_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-locales_4.9.3-11_all.deb
| DEBUG Using DebFile for a/build/gcc-4.9-multilib_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for
a/build/gcc-4.9-plugin-dev-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-plugin-dev_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-source_4.9.3-11_all.deb
| DEBUG Using GzipFile for a/build/gcc-4.9_4.9.3-11.diff.gz
| DEBUG Using DotDscFile for a/build/gcc-4.9_4.9.3-11.dsc
| DEBUG Using DotBuildinfoFile for
a/build/gcc-4.9_4.9.3-11_kfreebsd-amd64.buildinfo
| DEBUG validating sha1 checksums
and I'm pretty sure I used current Git master:
| grep -n -C2 irrelevant
/usr/lib/python3/dist-packages/diffoscope/comparators/deb.py
| 40-for member_name, member in container.get_members().items():
| 41-# Let's assume the name will end with .deb to avoid looking at
| 42:# too many irrelevant files
| 43-if not member_name.endswith('.deb'):
| 44- continue
Regards,
--
Steven Chamberlain
ste...@pyro.eu.org
signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] symlink permission bits on non-Linux
Hi, On linux, a symlink can only have permissions 0777 (lrwxrwxrwx) But on at least kfreebsd (maybe hurd?) there is no such limitation, and permissions are set like any regular file. That also means the umask is applied... and tar and dpkg-deb preserve this. This proves to be an issue for: * reproducible builds on kfreebsd, affected by user's umask * reproducing arch:all packages between linux<->kfreebsd * reproducing linux packages by cross-building from kfreebsd I think we should normalise symlinks' permissions to 0777, except GNU chmod can't do that! (chmod follows the symlink, and has no -h flag). Adding a -h (no dereference) option to chmod would allow dh_fixperms to use that. But (as pointed out in #759886) adding things there does not help packages not using debhelper, or other uses of tar. Would this be best added as a feature to tar, that dpkg-deb can use? Probably a new flag, that would apply --mode a=rwx only to symlinks. Or are there other ideas how to fix this? Thanks, Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] please don't build on archs that packages don't declare support for
Michael R. Crusoe wrote: > It looks fixed on my end; thanks! > https://tests.reproducible-builds.org/rb-pkg/testing/armhf/khmer.html Yep! khmer will now only build its arch-dep and -indep binaries on amd64. https://tests.reproducible-builds.org/unstable/armhf/index_not_for_us.html now has 255 packages, up from 249. So there weren't many cases of this really. https://tests.reproducible-builds.org/unstable/amd64/index_not_for_us.html now has 90 packages, up from 85. One new one is: https://buildd.debian.org/status/package.php?p=fso-frameworkd=unstable which didn't actually FTBFS on amd64, but it only produces armel and arch:all binaries, so it is quite reasonable that this is now only built and tested for reproducibility on armhf. Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] please don't build on archs that packages don't declare support for
Hi,
Holger Levsen wrote:
> khmer is being tried to build (on armhf), because it has "all" in the
> Architecture field
I think bin/reproducible_build.sh makes a too broad assumption, that
packages producing arch:all parts can build on "any" arch.
dpkg-buildpackage proceeds, I think, because there are arch-indep parts
that 'might' be able to build. But since this package has no separate
binary-indep target, debian/rules tries to build the arch-dep parts
and fails there.
It seems there was an attempt to fix this before:
https://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/commit/bin/reproducible_build.sh?id=c1157ef4d
but reverted:
https://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/commit/bin/reproducible_build.sh?id=cdbb4ce
https://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/commit/bin/reproducible_build.sh?id=89b4833f
I checked the archive of this mailing list but can't find the reason it
was reverted?
The code has been much refactored since then. Does my patch below seem
a neat way to try to fix this again? (Though I totally have not tested
it).
Where I set ARCHITECTURES="any", that will not be a regression over
current behaviour, but can be substituted for ARCHITECTURES="amd64" if
too many arch:all packages FTBFS on armhf.
From a759d049b1fd6deeb24985e57a3b6f4fa2e1f72b Mon Sep 17 00:00:00 2001
From: Steven Chamberlain <ste...@pyro.eu.org>
Date: Tue, 9 Feb 2016 13:02:13 +
Subject: [PATCH] reproducible: don't always build arch:all on "any" arch
If a package builds arch-dep parts on a restricted set of arches,
don't assume the arch-indep parts can build on "any" arch.
If a package *only* lists Architecture: all, keep the current behaviour:
(linux-)amd64 will most likely work; armhf is less likely but is nice
to try anyway.
---
bin/reproducible_build.sh | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/bin/reproducible_build.sh b/bin/reproducible_build.sh
index 4d5dd4b..7b11cef 100755
--- a/bin/reproducible_build.sh
+++ b/bin/reproducible_build.sh
@@ -525,8 +525,14 @@ check_suitability() {
# check whether the package is not for us...
local SUITABLE=false
local ARCHITECTURES=$(grep "^Architecture: " ${SRCPACKAGE}_*.dsc| cut
-d " " -f2- | sed -s "s# #\n#g" | sort -u)
+
+ # packages that are *only* arch:all can be tried on any arch
+ if [ "$ARCHITECTURES" = "all" ]; then
+ ARCHITECTURES="any"
+ fi
+
for arch in ${ARCHITECTURES} ; do
- if [ "$arch" = "any" ] || [ "$arch" = "$ARCH" ] || [ "$arch" =
"linux-any" ] || [ "$arch" = "linux-$ARCH" ] || [ "$arch" = "any-$ARCH" ] || [
"$arch" = "all" ] ; then
+ if [ "$arch" = "any" ] || [ "$arch" = "$ARCH" ] || [ "$arch" =
"linux-any" ] || [ "$arch" = "linux-$ARCH" ] || [ "$arch" = "any-$ARCH" ] ; then
SUITABLE=true
break
fi
Regards,
--
Steven Chamberlain
ste...@pyro.eu.org
signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Cross-building across architectures (reproducibly?)
Steven Chamberlain wrote: > using Helmut's tool, I've been able to rebootstrap a minimal Debian > linux-i386 chroot (445 binary packages[3]). These were cross-compiled > from source, by only running kfreebsd-amd64 binaries on a FreeBSD > kernel, and having some Arch:all packages installed. I've begun to analyze this now, using diffoscope to compare a stage1 debootstrapped chroot of linux-i386, from official Debian packages vs. my cross-built ones from kfreebsd-amd64. Unfortunately all the ELF binaries have differences, but most other files' contents are reproducible. The biggest problem was that the cross toolchain's linker used a different target ABI: │ │ │ -OS: Linux, ABI: 3.2.0 │ │ │ +OS: Linux, ABI: 2.6.32 and that leads to huge differences in the output binaries, such as not using .init_array/.fini_array sections. That may have consequences, such as if reproducible.debian.net jobs ever override uname to 2.6.x (Bug #806911), we may run into this problem there. umasks are applied to symlinks differently on kfreebsd than on linux: │ │ -lrwxrwxrwx root/root 0 2015-10-28 22:29:08 ./etc/os-release -> ../usr/lib/os-release │ │ +lrwxr-xr-x root/root 0 2015-10-28 22:29:08 ./etc/os-release -> ../usr/lib/os-release base-files seems to look at uname of the build architecture, rather than host arch it is building for: │ ├── ./usr/lib/os-release │ │ @@ -1,6 +1,6 @@ │ │ -PRETTY_NAME="Debian GNU/Linux stretch/sid" │ │ -NAME="Debian GNU/Linux" │ │ +PRETTY_NAME="Debian GNU/kFreeBSD stretch/sid" │ │ +NAME="Debian GNU/kFreeBSD" gzip embeds the version of a build-dependency, makeinfo. That's slightly annoying; it creates differences where there needn't have been otherwise: │ ├── ./usr/share/info/gzip.info.gz │ │ ├── gzip.info │ │ │ @@ -1,2827 +1,2827 @@ │ │ │ : 5468 6973 2069 7320 677a 6970 2e69 6e66 This is gzip.inf │ │ │ 0010: 6f2c 2070 726f 6475 6365 6420 6279 206d o, produced by m │ │ │ 0020: 616b 6569 6e66 6f20 7665 7273 696f 6e20 akeinfo version │ │ │ -0030: 352e 3220 6672 6f6d 2067 7a69 702e 7465 5.2 from gzip.te │ │ │ +0030: 362e 3020 6672 6f6d 2067 7a69 702e 7465 6.0 from gzip.te │ │ │ 0040: 7869 2e0a 0a54 6869 7320 6d61 6e75 616c xi Then I saw some some reproducibility issues that are already known (#774425 in dash, #806328 in xz-utils). Here are my chroots and diffoscope output, in case anyone is curious: http://kfreebsd.eu/rebootstrap/2016010100/native.tar.xz (12MiB) http://kfreebsd.eu/rebootstrap/2016010100/rebootstrap.tar.xz (12MiB) http://kfreebsd.eu/rebootstrap/2016010100/diff.xz (6MiB) Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Cross-building across architectures (reproducibly?)
Hi, I was going to introduce this topic, but it turned into an essay, so there is some background at: http://pyro.eu.org/writings/2016010100 The part most relevant to Reproducible Builds is: For arch-dependent binary packages, our latest breakthrough is that using Helmut's tool, I've been able to rebootstrap a minimal Debian linux-i386 chroot (445 binary packages[3]). These were cross-compiled from source, by only running kfreebsd-amd64 binaries on a FreeBSD kernel, and having some Arch:all packages installed. Ideally we'd like the native and cross-built binaries to be identical, but for many reasons they're currently not. They must build reproducibly in the first place, and Debian has made great progress there already. In the future, we'd like to be able to re-bootstrap more architectures, also in the reverse direction using linux to verify kfreebsd and hurd, and other combinations. For the Arch:all (architecture-independent) packages, there was some discussion recently on the reproducible-builds list[4]. I'd like those to be reproducible from more than one arch someday, with bonus points if that also works from different kernels. That only leaves bootloaders, kernels, and perhaps system firmware like coreboot; though they tend to cross-build quite easily actually, grub2 and kfreebsd package builds are mostly reproducible from Linux and kFreeBSD already. [3]: http://kfreebsd.eu/rebootstrap/2016010100/rebootstrap_i386_gcc5_nobiarch/ [4]: https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20151123/003992.html Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Juniper ScreenOS backdoor
Holger Levsen wrote: > https://github.com/hdm/juniper-cve-2015-7755/tree/master/firmware has links > to > the actual firmware images, I would appreciate if someone could throw them > against (my.)diffoscope.org and share the links…! Oh, didn't think of that! It may be a nice demo of diffoscope if it can do this, although it might not know how to disassemble this properly. I uploaded the firmwares here but I think something broke... it has been "in queue, please wait" for over an hour :( The files were 25MB each. https://try.diffoscope.org/quvzskqbuysh Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Juniper ScreenOS backdoor
Hi, One of the reproducible builds talk slides, showed a diff of OpenSSH before and after some off-by-one vulnerability was fixed. Here's a real-world malicious backdoor in Juniper ScreenOS's sshd: https://community.rapid7.com/servlet/JiveServlet/showImage/38-7376-36434/ssh.png The yellow highlighted string allows login as any user. Full article: https://community.rapid7.com/community/infosec/blog/2015/12/20/cve-2015-7755-juniper-screenos-authentication-backdoor Whilst this may have been added in source code, it was well-disguised in the disassembly and just 7 instructions long. I thought this was a good example of the current state-of-the-art, and why we'd like our binaries and eventually, installer and VM images reproducible IMHO. Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Juniper ScreenOS backdoor
Steven Chamberlain wrote: > I uploaded the firmwares here but I think something broke... it has been > "in queue, please wait" for over an hour :( The files were 25MB each. > https://try.diffoscope.org/quvzskqbuysh Okay, I did eventually finish. As suspected, diffoscope (or file or binutils) don't know how to disassemble this automatically, and neither do I. The ssh500*.bin files should be x86 code, already decompressed. The differences are too numerous for diffoscope to show all of it anyway. I guess Juniper didn't implement reproducible builds yet ;) Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Juniper ScreenOS backdoor
Hi, Chris Lamb wrote: > It actually finished in about 2 seconds but there was just a small bug: > > https://github.com/lamby/trydiffoscope/commit/3ed0ba502bf3f89d4c0599e3bcd390b3bb40f9f2 Thanks! And I was just about to point out the typo, but... https://github.com/lamby/trydiffoscope/commit/302190ac958b35fe95a0c2bc2d2a30f214822fc1 Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#805321: Bug#805321: debian-installer: builds unreproducible netboot images
Hi,
Cyril Brulebois wrote:
> I've cherry-picked 3 patches from there onto master locally and I'm
> currently running diffoscope to see how that goes (and it's taking
> ages…):
I'm guessing the initrd would differ if the Linux tool to generate it
stores timestamps. If its compressed size varies much due to those
differences, the .iso block numbers may vary as a result.
> FWIW, I'm not exactly entirely convinced by the exporting of the
> SOURCE_DATE_EPOCH variable from debian/rules; all other variables have
> been passed without exporting so I'm wondering if we shouldn't adapt
> this to behave like other variables, reducing possible surprise for
> users.
Just to explain that -- if it's defined in the environment, it requires
no special handling and doesn't need to be (re-)exported. I think this
is maybe the case now for dpkg-buildpackage in sid?
If the dpkg-buildpackage environment doesn't have SOURCE_DATE_EPOCH
(e.g. jessie), debian/rules sets it to the correct value, and so must
export that.
Or (for jessie or sid), in case build/Makefile is used directly, outside
of a package build, we set SOURCE_DATE_EPOCH to a dummy value ("now") if
undefined (since ../debian/changelog may not exist), which we need
when calling makefs from within that Makefile. We export it for use by
gen-tarball to avoid duplication there.
Regards,
--
Steven Chamberlain
ste...@pyro.eu.org
signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] reproducible builds of FreeBSD in a chroot on Linux
Hi Holger, Holger Levsen wrote: With this, http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/tree/bin/reproducible_freebsd.sh gets as far as https://jenkins.debian.net/view/reproducible/job/reproducible_freebsd/7/console where stage 2.1: cleaning up the object tree fails on make buildworld, because /srv/workspace/chroots/freebsd- .v1adN6Qo/freebsd/lib/libc/tests does not exist. `mktemp freebsd-` on FreeBSD would result in random characters being appended, resulting in freebsd-.v1adN6Qo as above. `mktemp -d -t freebsd-` should replace the X's with random characters, same as GNU mktemp. But it doesn't seem to have done that. Are you sure that your RSSH command is sending switches -d and -t correctly, or do you need a -- or extra quotes? Take a look in /srv/workspace/chroots/ and see if mktemp has perhaps created a file instead of a directory? Regards, -- Steven Chamberlain ste...@pyro.eu.org signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#791584: kfreebsd-source-10.1: umask setting can affect build
Package: kfreebsd-source-10.1 Version: 10.1~svn274115-6 Severity: wishlist Tags: patch pending User: reproducible-builds@lists.alioth.debian.org Usertags: timestamps Hi, The kfreebsd-10 (kernel) packaging distributes a source tarball in kfreebsd-source, a binary arch-indep package. Files and directories in the tarball are owned by root:src, but permissions depend on the package builder's umask setting. To ensure reproducibility, I guess we could chmod before building the tarball: --- debian/rules(revision 5716) +++ debian/rules(working copy) @@ -170,6 +170,7 @@ mkdir -p $(SOURCE_PACKAGE)/usr/src chown -R root.src $(SRC_DIR) + chmod -R go+r-w $(SRC_DIR) # Clamp timestamps to be no newer than last changelog entry, see # https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball find $(SRC_DIR) -newermt $(BUILD_DATE) -print0 \ -- System Information: Debian Release: jessie/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: kfreebsd-amd64 (x86_64) Kernel: kFreeBSD 9.0-2-amd64-xenhvm-ipsec Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#791584: kfreebsd-source-10.1: umask setting can affect build
Steven Chamberlain wrote: [...] I guess we could chmod before building the tarball: On second thoughts, that's not as efficient as using tar's --mode option so I'll use that. The example given on the Debian Wiki page is more comprehensive than my chmod anyway. --- debian/rules(revision 5717) +++ debian/rules(revision 5719) @@ -176,8 +176,11 @@ | xargs -0r touch --no-dereference --date=$(BUILD_DATE) # Create tarball with files sorted in a stable order, see # https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs + # and normalised file permissions regardless of umask, see + # https://wiki.debian.org/ReproducibleBuilds/VaryingPermissionsInTarballs find $(SRC_DIR) -print0 | LC_ALL=C sort -z \ | tar --null --no-recursion --files-from - \ + --mode=go=rX,u+rw,a-s \ -cJf $(SOURCE_PACKAGE)/usr/src/$(SRC_TAR) touch install-indep-stamp Regards, -- Steven Chamberlain ste...@pyro.eu.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#782878: [debhelper-devel] Bug#782879 + Bug#782878: lib{test-log4perl, scalar-defer}-perl: please make the build reproducible
Hi, Russ Allbery wrote: My feeling is that the date in the man page serves a useful purpose for the end user by communicating some idea of the staleness of the documentation and the recentness of the last release of the software. While this isn't a huge deal, it does feel somewhat less than ideal to lose that data. Replacing it with the last modification date of the Debian package isn't perfect, but it's fairly reasonable. I'm suffering this issue in a different context, which is a binary package that ships a tarball of the kfreebsd (kernel) source. Some of those files are patched by dpkg-source. From #759404 I can see why dpkg-source can't really help with this. It would be a shame to lose / reset all the timestamps, because: * it's useful to know how old a file is, if Debian didn't patch it, * installing the package, extracting files with newer timestamps, could mean some systems for backup or deduplication must treat the files as though they are new/changed, * in a new package revision, if the files in a particular .deb didn't change, it would be nice if the data.tar.xz didn't change; this would be helpful to future work on .deb deltas or deduplication. I suggest to only 'clamp' timestamps to the latest entry in debian/changelog. I think only timestamps newer than this are likely an issue for reproducibility. Older timestamps are potentially still useful. Regards, -- Steven Chamberlain ste...@pyro.eu.org ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#786615: kfreebsd-10: source tarballs have unreproducible file order
Package: src:kfreebsd-10 Version: 10.1~svn274115-4 Severity: wishlist Tags: patch pending User: reproducible-builds@lists.alioth.debian.org Usertags: fileordering Hi, The kfreebsd-10 (kernel) packaging creates two source tarballs: * orig.tar.xz, updated by the package maintainer for each new upstream version * in kfreebsd-source, a binary arch-indep package The ordering of files is uncertain when these tarballs are created, perhaps especially on the ZFS filesystem. The attached patch will fix that, so that independently built tarballs don't differ needlessly, and any real difference in their content is easy to identify. -- System Information: Debian Release: jessie/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: kfreebsd-amd64 (x86_64) Kernel: kFreeBSD 9.0-2-amd64-xenhvm-ipsec Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Create tar archive with deterministic file order
Hi,
The reproducible.d.n Jenkins jobs revealed a file ordering issue when we
construct kfreebsd-source-10.1.tar.xz from src:kfreebsd-10 this way:
tar cfJ $(SOURCE_PACKAGE)/usr/src/$(SRC_TAR) $(SRC_DIR)
What would be a preferred way to build a tar archive with deterministic
file order?
Firstly let me point out that:
$ find foo/ -type f -print0 | sort -z | xargs -0 tar -cvf foo.tar
is risky as the list of filenames could overflow what xargs can supply
to a single invocation of tar. In that case it would invoke tar again
and clobber the previous output.
Using the xargs -n1 option we can simulate that happening:
$ mkdir foo/
$ touch foo/{a,b,c}
$ find foo/ -type f -print0 | sort -z | xargs -n1 -0 tar -cvf foo.tar
foo/a
foo/b
foo/c
$ tar -tvf foo.tar
-rw-r--r-- steven/steven 0 2015-02-14 20:05 foo/c
I expected the following to work; it does concatenate the output of tar
into a single file. But at least with GNU tar, the first invocation of
tar outputs an end-of-file marker, so archives can't be concatenated this
way. (It might also have not been reproducible, if the archive split
point can vary between systems). Listing the archive's contents only
returns the first entry:
$ find foo/ -type f -print0 | xargs -n1 -0 tar -cvf - foo.tar
foo/a
foo/b
foo/c
$ tar -tvf foo.tar
-rw-r--r-- steven/steven 0 2015-02-14 20:05 foo/a
So I came to this:
$ rm -f foo.tar find foo/ -type f -print0 | xargs -n1 -0 tar -rvf foo.tar
using the --append (-r) option of GNU tar. It works, but that's not
really ideal, as I'd prefer to pipe the output through xz before writing
anything to disk. The -r option can't be used with stdout or -J.
Finally I ended up with this:
$ find foo/ -type f -print0 filelist
$ tar -Jcvf foo.tar.xz -T filelist --null
Does that seem like the neatest way, or do you have better suggestions?
(I thought this problem would be quite common, so I could add it to the
Wiki FAQ).
Thanks!
Regards,
--
Steven Chamberlain
ste...@pyro.eu.org
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
