Re: Bug#894441: dpkg-buildpackage: SOURCE_DATE_EPOCH must ignore bin-nmu changelog entries. Breaks M-A:same

[Guillem Jover]

Control: reassign -1 buildd.debian.org

Hi!

On Thu, 2018-04-05 at 17:43:58 +0200, Jean-Michel Vourgère wrote:
> On Friday, 30 March 2018 15:02:31 CEST Chris Lamb wrote:
> > [ https://lists.debian.org/debian-security/2017/05/msg00011.html ]
> 
> On Friday, 30 March 2018 20:15:33 CEST Sven Joachim wrote:
> > [ https://bugs.debian.org/843773 ]
> 
> Thanks a lot guys for pointing out that issue!
> 
> Basically, when doing bin-nmus, we really want to bump the mtime of the 
> distributed files. Not doing so results in some backups programs (rsync...) 
> to 
> loose updates. Other programs restarting services on libraries updates 
> (needrestart...) would also be affected.
> 
> 
> So, during compilation:
> SOURCE_DATE_EPOCH must ignore bin-nmu changelog entries
> because it breaks Multi-Arch:same on bin-nmu.
> 
> During dpkg-deb (:
> SOURCE_DATE_EPOCH must *not* ignore bin-nmu changelog entries
> because it would break software relying on files mtime.
> 
> Doh!
> 
> In https://bugs.debian.org/843773#75 Ian Jackson propose to introduce a 
> BUILD_DATE_EPOCH (= time of sbuild binnmu invocation) be prefered over 
> SOURCE_DATE_EPOCH by dpkg-deb.
> 
> That would work, wouldn't it?

Please, see my reply at . This is
really a fundamental problem with binNMUs+multiarch-refcounting or how
they are being issued. :)

Thanks,
Guillem


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#869184: dpkg: source uploads including _amd64.buildinfo cause problems

[Guillem Jover]

On Thu, 2018-03-01 at 15:22:30 +, Holger Levsen wrote:
> On Wed, Jan 24, 2018 at 04:05:39PM +0100, Salvatore Bonaccorso wrote:
> > Any news regarding this proposal from Ansgar? We were biten now
> > several times already by this (e.g. php update, curl via
> > security.d.o).
> 
> Guilem, what's your stance on this bug?

I think it should be fixed. I've just not come up with something that
would seem a generic/clean way to do that. :(

> We (reproducible builds) really dont want "our" tools (=.buildinfo files)
> to cause grief to other teams in Debian, and especially not on a regular
> basis... so as a first step to fix this, I'd like to collect opinions
> how to best fix this issue here.

The problem got introduced when I proposed changing the filename
format, and dropping the annoying timestamp. I also though there was
talk at some point initially about DAK renaming those files to cope
with possible multiple uploads if the conflicting names?

Renaming the arch buildinfo to _source.buildinfo seems wrong, and even
then I'm not sure how to cleanly transfer that knowledge from
dpkg-buildpackage to dpkg-genbuildinfo.

I guess, the ideal solution would be to qualify the buildinfo file
with the builder user and hostname, because that in a way denotes the
build environment. But that seems like too much leakage. As in:

  pkgfoo_1.0.0-1_mips64el_username@hostname.buildinfo

Perhaps just using the maintainer email address might be enough though,
the one from the -m option or from the changelog, which AFAIR buildds
do set? But this seems like it can produce quite ugly filenames:

  
pkgfoo_1.0.0-1_mips64el_buildd_mipsel-mipsel-sil...@buildd.debian.org.buildinfo

not to mention that both of these "break" the conventional pattern, which
is still used by things like the debian/files parser and injector.

Perhaps the simplest and more correct might be to name it using
something like source+amd64 as the arch name, which seems like a
dubious arch, but at least is accurate and might be trivial to
implement, taking care of not ending up with such fake arch in
unexpected places…

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#873937: dpkg: should include information about the used kernel in .buildinfo files

[Guillem Jover]

Hi!

On Wed, 2017-09-06 at 19:18:33 -0700, Vagrant Cascadian wrote:
> On 2017-09-02, Holger Levsen wrote:
> > On Fri, Sep 01, 2017 at 04:51:55PM +0200, Guillem Jover wrote:
> > > In addition to the above, I'm actually somewhat uncomfortable with this
> > > request, as it looks like a massive privacy leak. Compared to package
> > > lists and versions, which are actually requested by the package being
> > > built and might not have anything to do with the main system this
> > > build was being run on (say a chroot for example), or might get deleted
> > > immediately after the build. The kernel tends to be a system-wide
> > > resource, that even if upgraded does not mean it will be running (until
> > > a reboot).
> >
> > on reflection I agree that the privacy implications are too bad.
> 
> The including the build path also has privacy implications, but it can
> be disabled from inclusion in .buildinfo, no?  What about including the
> kernel if something like DEB_BUILD_OPTS="buildinfo=+kernel" ?

Ah good point, yeah, I have no problem with adding this as an option
that is disabled by default. Attached a tentative patch doing that.

Thanks,
Guillem
From ca1a160894ecd1d1b1bc71228540a117e1a0a9a1 Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Sun, 10 Sep 2017 16:18:15 +0200
Subject: [PATCH] dpkg-genbuildinfo: Add a new Build-Kernel-Version field
 disabled by default

Packages intended to be built in a generic way must never rely on the
currently running kernel on the build system (an exception could be an
optimization rebuild using the current system as the reference baseline).

But to be able to detect when a package might not be reproducible due to
varying kernel information it is still useful to be able to record this
information. Although that information can be very sensitive.

We add a new Build-Kernel-Version field which needs to be enabled
explicitly by the builder so that it gets included in the .buildinfo
file.

Closes: #873937
---
 man/deb-buildinfo.man  | 6 ++
 man/dpkg-genbuildinfo.man  | 6 ++
 scripts/Dpkg/Control/FieldsCore.pm | 6 +-
 scripts/dpkg-genbuildinfo.pl   | 8 
 4 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/man/deb-buildinfo.man b/man/deb-buildinfo.man
index 3b184241e..5013aa047 100644
--- a/man/deb-buildinfo.man
+++ b/man/deb-buildinfo.man
@@ -135,6 +135,12 @@ The date the package was built.
 It must be in the same format as the date in a \fBdeb\-changelog\fP(5)
 entry.
 .TP
+.BR Build\-Kernel\-Version: " \fIbuild-kernel-version\fP"
+The release and version (in an unspecified format) of the kernel running
+on the build system.
+This field is only going to be present if the builder has explicitly
+requested it, to avoid leaking possibly sensitive information.
+.TP
 .BR Build\-Path: " \fIbuild-path\fP"
 The absolute build path, which correspond to the unpacked source tree.
 This field is only going to be present if the vendor has whitelisted it
diff --git a/man/dpkg-genbuildinfo.man b/man/dpkg-genbuildinfo.man
index aaf54a38a..f97f4d316 100644
--- a/man/dpkg-genbuildinfo.man
+++ b/man/dpkg-genbuildinfo.man
@@ -104,6 +104,12 @@ rather than
 (\fBdpkg\-genbuildinfo\fP needs to find these files so that it can include
 their sizes and checksums in the \fB.buildinfo\fP file).
 .TP
+.BI \-\-always\-include\-kernel
+By default, the \fBBuild\-Kernel-Version\fR field will not be written out.
+
+Specify this option (since dpkg 1.19.0) to always write a
+\fBBuild\-Kerel\-Version\fR field when generating the \fB.buildinfo\fR.
+.TP
 .BI \-\-always\-include\-path
 By default, the \fBBuild\-Path\fR field will only be written if the current
 directory starts with a whitelisted pattern.
diff --git a/scripts/Dpkg/Control/FieldsCore.pm b/scripts/Dpkg/Control/FieldsCore.pm
index 78e59d393..7c9ec47e7 100644
--- a/scripts/Dpkg/Control/FieldsCore.pm
+++ b/scripts/Dpkg/Control/FieldsCore.pm
@@ -143,6 +143,9 @@ our %FIELDS = (
 'Build-Essential' => {
 allowed => ALL_PKG,
 },
+'Build-Kernel-Version' => {
+allowed => CTRL_FILE_BUILDINFO,
+},
 'Build-Origin' => {
 allowed => CTRL_FILE_BUILDINFO,
 },
@@ -479,7 +482,8 @@ our %FIELD_ORDER = (
 qw(Format Source Binary Architecture Version
 Binary-Only-Changes),
 @checksum_fields,
-qw(Build-Origin Build-Architecture Build-Date Build-Path
+qw(Build-Origin Build-Architecture Build-Date Build-Kernel-Version
+   Build-Path
Installed-Build-Depends Environment),
 ],
 CTRL_FILE_CHANGES() => [
diff --git a/scripts/dpkg-genbuildinfo.pl b/scripts/dpkg-genbuildinfo.pl
index 3682f2f02..e6350bba2 100755
--- a/scripts/dpkg-genbuildinfo.pl
+++ b/scripts/dpkg-genbuildinfo.pl
@@ -61,6 +61,7 @@ my $outputfile;
 my $stdout =

Re: source-only builds and .buildinfo

[Guillem Jover]

Hi!

[ Just a very quick reply, will go over the other mails during the week. ]

On Wed, 2017-05-24 at 13:58:00 +, Ximin Luo wrote:
> Also the man page for dpkg-buildpackage is out-of-date:
> 
>6. Unless a source-only build has been requested, it runs the
> buildinfo hook and calls dpkg-genbuildinfo to generate a .buildinfo
> file.  Several dpkg-buildpackage options are forwarded to dpkg-genbuildinfo.
> 
> and also later:
> 
>   The current hook-name supported are:
>   init preclean source build binary changes postclean check
>   sign done
> 
> missing out "buildinfo", and indeed if I run "dpkg-buildpackage
> --hook-buildinfo=true" the buildinfo file still gets generated.

Yes, as I mentioned on
 this is
something I've noticed now several times, but forgot to fix. I did so
the other day and have queued it for a future 1.18.25 release.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#855282: debsign: support .buildinfo files

[Guillem Jover]

Hi!

On Mon, 2017-03-13 at 00:02:26 -0400, James McCoy wrote:
> On Mon, Mar 06, 2017 at 11:45:20PM -0500, James McCoy wrote:
> > This was merged and uploaded in 2.17.2.  However, now I see that
> > buildinfo files may be arch-qualified even with a _source.changes (e.g.,
> > by using "sbuild -A --source-only-changes").  That's not currently
> > handled properly since "debsign foo_ver_source.changes" expects
> > "foo_ver_source.buildinfo" to exist.
> > 
> > Should debsign be deriving the path for the buildinfo from the contents
> > of the changes file?

Sorry for not chiming in sooner!

> I've changed all of the "child file" handling to use this approach in
> 4a4238dbf1e789e998cf047ea0f006e982fba56b.  That seems cleaner than
> debsign replicating logic to determine the names of related files.

Yeah, thanks, that sounds best. Not even scripts in dpkg-dev are
assuming any specific name for the buildinfo file, they are always
getting it from the .changes file.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Moving towards a deb-buildinfo(5) Format 1.0

[Guillem Jover]

Hi again!

[ I just had this drafted around, and though I just update and send it
  out as a status update. ]

On Sun, 2017-02-19 at 18:52:16 +0100, Guillem Jover wrote:
> On Sat, 2016-11-12 at 19:04:53 +0100, Guillem Jover wrote:
> > As I've mentioned elsewhere, I've noticed several things with the
> > current .buildinfo format, even after the cleanup pre-merge, that
> > I'd like to fix or change so that we can hopefully reach Format 1.0.
> 
> Ok, let's see what's the current status:

[…]

I forgot to mention two curretly pending issues, not listed previously:

* An error in dpkg-genbuildinfo caused by arch-qualified dependencies
  on a virtual package.

  This was in dpkg 1.18.23. An implementation detail though, even if
  unfortunate.

* Broken dependency recursor in dpkg-genbuildinfo

  The dependency traversal in dpkg-genbuildinfo is pretty much broken
  for many/most Multi-Arch cases. But the same applies to
  dpkg-checkbuilddeps and the Dpkg::Deps perl module. This needs major
  rework and is in a way a pre-existing problem, and in any case an
  implementation detail.

> > I'll probably do some of the fixes already and bump Format to 0.2
> > and after the discussion settles we can perhaps do a 0.3, and see how
> > it goes, and iterate until it looks good, at which point we'd declare
> > it 1.0, ideally before the freeze. :)
> 
> So given the above, I've queued a minimal change declaring the format
> 1.0 for dpkg 1.18.23 or .24, please shout if you see any additional
> problem or blocker.

This has happened now, given that the above are implementation details,
and do not really affect the format.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#855282: debsign: support .buildinfo files

[Guillem Jover]

Hi!

On Fri, 2017-02-17 at 06:08:25 +0100, Guillem Jover wrote:
> On Thu, 2017-02-16 at 17:23:00 +, Ximin Luo wrote:
> > Control: tags + patch
> 
> > I've done an initial implementation here:
> > 
> > https://anonscm.debian.org/cgit/collab-maint/devscripts.git/log/?h=pu/debsign-buildinfo
> > 
> > Please review!
> 
> I think something like the attached patch on top of your branch HEAD
> is also needed.

Here's another patch to support the finalized format 1.0 sitting now
in dpkg's git master, pending upload to unstable.

Thanks,
Guillem
From d21172ba5d15f920929892e72ccc7bd83024628f Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Tue, 28 Feb 2017 00:13:52 +0100
Subject: [PATCH] Add support for finalized .buildinfo format 1.0

---
 scripts/debsign.sh   | 2 +-
 scripts/dscverify.pl | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/debsign.sh b/scripts/debsign.sh
index 4b2b22cb..2eb23df5 100755
--- a/scripts/debsign.sh
+++ b/scripts/debsign.sh
@@ -462,7 +462,7 @@ fixup_control() {
 }
 
 fixup_buildinfo() {
-fixup_control '$major != 0 or $minor > 2' dsc buildinfo "$@"
+fixup_control '($major != 0 or $minor > 2) and ($major != 1 or $minor > 0)' dsc buildinfo "$@"
 }
 
 fixup_changes() {
diff --git a/scripts/dscverify.pl b/scripts/dscverify.pl
index 381ebff5..45f2c605 100755
--- a/scripts/dscverify.pl
+++ b/scripts/dscverify.pl
@@ -208,7 +208,8 @@ sub process_file {
 	$major += 0;
 	$minor += 0;
 	if ($file =~ /\.changes$/ and ($major != 1 or $minor > 8) or
-	$file =~ /\.buildinfo$/ and ($major != 0 or $minor > 2)) {
+	$file =~ /\.buildinfo$/ and (($major != 0 or $minor > 2) and
+	 ($major != 1 or $minor > 0))) {
 	xwarn "$file is an unsupported format: $format\n";
 	return;
 	}
-- 
2.12.0.rc1.440.g5b76565f74

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Moving towards a deb-buildinfo(5) Format 1.0

[Guillem Jover]

On Sun, 2017-02-19 at 11:39:28 -0800, Vagrant Cascadian wrote:
> On 2017-02-19, Guillem Jover wrote:
> >> * .buildinfo files are not generated when creating source-only uploads
> >
> > Fixed. Now always generated.
> 
> On a related note, is it currently possible to create a .buildinfo with
> both the source and binary, but a corresponding .changes with only the
> source?

Yes, in the same way it's been possible to do so for source-only
uploads up-to-now when using just dpkg-buildpackage:

  $ dpkg-buildpackage --changes-option=-S

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Moving towards a deb-buildinfo(5) Format 1.0

[Guillem Jover]

Hi!

On Sat, 2016-11-12 at 19:04:53 +0100, Guillem Jover wrote:
> As I've mentioned elsewhere, I've noticed several things with the
> current .buildinfo format, even after the cleanup pre-merge, that
> I'd like to fix or change so that we can hopefully reach Format 1.0.

Ok, let's see what's the current status:

> Some of the issues, that bother me:
> 
> * .buildinfo files are not currently signed

Fixed. Pending debsign (from devscipts) doing the same.

> * .buildinfo filename

Fixed. Now they use the same format as .changes files.

> * dpkg-genbuildinfo injects itself into debian/files

I think this is fine for now, we can always revisit later on. In any
case this is an implementation detail not affecting the .buildinfo
format.

> * .buildinfo files are not generated when creating source-only uploads

Fixed. Now always generated.

> * .buildinfo has some issues when including .dsc information
> 
>   Only the .dsc file is referenced not all of its contents, it might
>   be better to match .changes logic here. Also the “source”
>   pseudo-architecture does not get added to the Architecture field.
>   I'll just do at least the latter, I'm open for discussion on the
>   former.

Partially fixed. The source is now included in the Architecture
field. The inclusion is not recursive, but I think this is fine,
and we should be able to add them if we deem it important in the
future w/o breaking the format.

> * Some of the environment variables seem superfluous or leaks

Ignored. This is valuable information, so it seems fine. Also new
variables are alos tracked now.

> I'll probably do some of the fixes already and bump Format to 0.2
> and after the discussion settles we can perhaps do a 0.3, and see how
> it goes, and iterate until it looks good, at which point we'd declare
> it 1.0, ideally before the freeze. :)

So given the above, I've queued a minimal change declaring the format
1.0 for dpkg 1.18.23 or .24, please shout if you see any additional
problem or blocker.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#855282: debsign: support .buildinfo files

[Guillem Jover]

Hi!

On Thu, 2017-02-16 at 17:23:00 +, Ximin Luo wrote:
> Control: tags + patch

> I've done an initial implementation here:
> 
> https://anonscm.debian.org/cgit/collab-maint/devscripts.git/log/?h=pu/debsign-buildinfo
> 
> Please review!

I think something like the attached patch on top of your branch HEAD
is also needed.

Thanks,
Guillem
From 8ccb601c3e3f1cb16db923b8464e8a73b4cf03df Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Fri, 17 Feb 2017 06:04:27 +0100
Subject: [PATCH] Improve .buildinfo support

---
 scripts/debsign.sh   | 19 +--
 scripts/dscverify.pl |  7 ---
 2 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/scripts/debsign.sh b/scripts/debsign.sh
index ede3aa79..4b2b22cb 100755
--- a/scripts/debsign.sh
+++ b/scripts/debsign.sh
@@ -418,7 +418,7 @@ fixup_control() {
 '" \$file=\"$child\"; \$md5=\"$md5\"; "'
 '" \$sha1=\"$sha1\"; \$sha256=\"$sha256\"; "'
 $size=(-s $file); ($base=$file) =~ s|.*/||;
-$infiles=0; $insha1=0; $insha256=0; $format="";
+$infiles=0; $inmd5=0; $insha1=0; $insha256=0; $format="";
 }
 if(/^Format:\s+(.*)/) {
 	$format=$1;
@@ -429,19 +429,26 @@ fixup_control() {
 	die "Unsupported .$parenttype format: $format\n"
 	if('"$filter_out"');
 }
-/^Files:/i && ($infiles=1,$insha1=0,$insha256=0);
-if(/^Checksums-Sha1:/i) {$insha1=1;$infiles=0;$insha256=0;}
+/^Files:/i && ($infiles=1,$inmd5=0,$insha1=0,$insha256=0);
+if(/^Checksums-Sha1:/i) {$insha1=1;$infiles=0;$inmd5=0;$insha256=0;}
 elsif(/^Checksums-Sha256:/i) {
-	$insha256=1;$infiles=0;$insha1=0;
+	$insha256=1;$infiles=0;$inmd5=0;$insha1=0;
+} elsif(/^Checksums-Md5:/i) {
+	$inmd5=1;$infiles=0;$insha1=0;$insha256=0;
 } elsif(/^Checksums-.*?:/i) {
 	die "Unknown checksum format: $_\n";
 }
-/^\s*$/ && ($infiles=0,$insha1=0,$insha256=0);
+/^\s*$/ && ($infiles=0,$inmd5=0,$insha1=0,$insha256=0);
 if ($infiles &&
 	/^ (\S+) (\d+) (\S+) (\S+) \Q$base\E\s*$/) {
 	$_ = " $md5 $size $3 $4 $base\n";
 	$infiles=0;
 }
+if ($inmd5 &&
+	/^ (\S+) (\d+) \Q$base\E\s*$/) {
+$_ = " $md5 $size $base\n";
+$inmd5=0;
+}
 if ($insha1 &&
 	/^ (\S+) (\d+) \Q$base\E\s*$/) {
 	$_ = " $sha1 $size $base\n";
@@ -455,7 +462,7 @@ fixup_control() {
 }
 
 fixup_buildinfo() {
-fixup_control '$major < 1 and $minor < 2' dsc buildinfo "$@"
+fixup_control '$major != 0 or $minor > 2' dsc buildinfo "$@"
 }
 
 fixup_changes() {
diff --git a/scripts/dscverify.pl b/scripts/dscverify.pl
index 8bc0857c..381ebff5 100755
--- a/scripts/dscverify.pl
+++ b/scripts/dscverify.pl
@@ -198,7 +198,7 @@ sub process_file {
 	}
 }
 
-if ($file =~ /\.changes$/ and $out =~ /^Format:\s*(.*)$/mi) {
+if ($file =~ /\.(changes|buildinfo)$/ and $out =~ /^Format:\s*(.*)$/mi) {
 	my $format = $1;
 	unless ($format =~ /^(\d+)\.(\d+)$/) {
 	xwarn "$file has an unrecognised format: $format\n";
@@ -207,7 +207,8 @@ sub process_file {
 	my ($major, $minor) = split /\./, $format;
 	$major += 0;
 	$minor += 0;
-	unless ($major == 1 and $minor <= 8) {
+	if ($file =~ /\.changes$/ and ($major != 1 or $minor > 8) or
+	$file =~ /\.buildinfo$/ and ($major != 0 or $minor > 2)) {
 	xwarn "$file is an unsupported format: $format\n";
 	return;
 	}
@@ -220,7 +221,7 @@ sub process_file {
 }
 
 my @checksums = map { split /\n/ } $out =~ /^Checksums-(\S+):\s*\n/mgi;
-@checksums = grep {!/^Sha(1|256)$/i} @checksums;
+@checksums = grep {!/^(Md5|Sha(1|256))$/i} @checksums;
 if (@checksums) {
 	xwarn "$file contains unsupported checksums:\n"
 	. join (", ", @checksums) . "\n";
-- 
2.11.0.483.g087da7b7c

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#853274: lintian: Please recurse into .buildinfo to find .debs

[Guillem Jover]

Package: lintian
Version: 2.5.50
Severity: wishlist
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Hi!

Now that .buildinfo files are always generated, even for source-only
uploads, it would be really nice if lintian would recurse into them to
find references to any .deb that is listed in .buildinfo but not in
.changes. One way to generate such source-only uploads after a
full-build is:

  $ dpkg-buildpackage --changes-option=-S

Which does a full-build, so that one can be sure the thing builds at
all, but will just upload the source package and the .buildinfo file.

Otherwise right now one has to call lintian like this:

  $ lintian p_v_a.changes p_v_*.deb

Or perhaps also supporting to call lintian like this:

  $ lintian p_v_a.buildinfo

would be nice. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#852801: devscripts: Please add support for signing/signed .buildinfo

[Guillem Jover]

Source: devscripts
Source-Version: 2.17.0
Severity: important
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Hi!

The .buildinfo files were supposed to be signed, but dpkg-buildpackage
didn't do that until dpkg 1.18.19. Even then, when we sign sources and
those get referenced in the .buildinfo file, their checksums will not
match as they have been changed.

I've prepared a patch for dscverify to test the new dpkg, but debsign
is still pending. Patch attached, please review.

Thanks,
Guillem
From 1579878c73e248f89d5619d893dab450fc6344fb Mon Sep 17 00:00:00 2001
From: Guillem Jover 
Date: Fri, 27 Jan 2017 14:02:44 +0100
Subject: [PATCH] dscverify: Add support for .buildinfo files

Signed-off-by: Guillem Jover 
---
 scripts/dscverify.1  | 11 +++
 scripts/dscverify.pl | 10 +-
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/scripts/dscverify.1 b/scripts/dscverify.1
index 48b41a25..1082147a 100644
--- a/scripts/dscverify.1
+++ b/scripts/dscverify.1
@@ -2,15 +2,17 @@
 .SH NAME
 dscverify \- verify the validity of a Debian package
 .SH SYNOPSIS
-\fBdscverify\fR [\fB\-\-keyring \fIkeyring\fR] ... \fIchanges_or_dsc_filename\fR ...
+\fBdscverify\fR [\fB\-\-keyring \fIkeyring\fR] ... \fIchanges_or_buildinfo_or_dsc_filename\fR ...
 .SH DESCRIPTION
 \fBdscverify\fR checks that the GPG signatures on the given
-\fI.changes\fR or \fI.dsc\fR files are good signatures made by keys in
+\fI.changes\fR, \fI.buildinfo\fP or \fI.dsc\fR files are good signatures
+made by keys in
 the current Debian keyrings, found in the \fIdebian-keyring\fR
 and \fIdebian-maintainers\fR
 packages.  (Additional keyrings can be specified using the
 \fB--keyring\fR option any number of times.)  It then checks that the
-other files listed in the \fI.changes\fR or \fI.dsc\fR files have the
+other files listed in the \fI.changes\fR, \fI.buildinfo\fP or \fI.dsc\fR
+files have the
 correct sizes and checksums (MD5 plus SHA1 and SHA256 if the latter are
 present).  The exit status is 0 if there are no problems and non-zero
 otherwise.
@@ -28,7 +30,8 @@ first option given on the command-line.
 .TP
 \fB\-\-nosigcheck\fR, \fB\-\-no\-sig\-check\fR, \fB-u\fR
 Skip the signature verification step. That is, only verify the sizes and
-checksums of the files listed in the \fI.changes\fR or \fI.dsc\fR files.
+checksums of the files listed in the \fI.changes\fR, \fI.buildinfo\fP or
+\fI.dsc\fR files.
 .TP
 \fB\-\-verbose\fR
 Do not suppress GPG output.
diff --git a/scripts/dscverify.pl b/scripts/dscverify.pl
index 97600856..8bc0857c 100755
--- a/scripts/dscverify.pl
+++ b/scripts/dscverify.pl
@@ -46,7 +46,7 @@ my $havegpg = first { !system('sh', '-c', "command -v $_ >/dev/null 2>&1")  } qw
 
 sub usage {
 print <<"EOF";
-Usage: $progname [options] dsc-or-changes-file ...
+Usage: $progname [options] changes-or-buildinfo-dsc-file ...
   Options: --help  Display this message
--version   Display version and copyright information
--keyring 
@@ -213,7 +213,7 @@ sub process_file {
 	}
 }
 
-my @spec = map { split /\n/ } $out =~ /^Files:\s*\n((?:[ \t]+.*\n)+)/mgi;
+my @spec = map { split /\n/ } $out =~ /^(?:Checksums-Md5|Files):\s*\n((?:[ \t]+.*\n)+)/mgi;
 unless (@spec) {
 	xwarn "no file spec lines in $file\n";
 	return;
@@ -342,7 +342,7 @@ sub process_file {
 
 	close FILE;
 
-	if ($filename =~ /\.dsc$/ && $verify_sigs) {
+	if ($filename =~ /\.(?:dsc|buildinfo)$/ && $verify_sigs) {
 	$sigcheck = check_signature $filename, @rings;
 	if ($sigcheck) {
 		xwarn "$filename failed signature check:\n$sigcheck";
@@ -358,7 +358,7 @@ sub process_file {
 }
 
 sub main {
-@ARGV or xdie "no .changes or .dsc files specified\n";
+@ARGV or xdie "no .changes, .buildinfo or .dsc files specified\n";
 
 my @rings;
 
@@ -416,7 +416,7 @@ sub main {
 	'verbose' => \$verbose,
 ) or do { usage; exit 1 };
 
-@ARGV or xdie "no .changes or .dsc files specified\n";
+@ARGV or xdie "no .changes, .buildinfo or .dsc files specified\n";
 
 @rings = get_rings @rings if $use_default_keyrings and $verify_sigs;
 
-- 
2.11.0

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Moving towards a deb-buildinfo(5) Format 1.0

[Guillem Jover]

Hi!

On Sun, 2016-11-13 at 14:21:45 +0100, Johannes Schauer wrote:
> Also see:
> 
> https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles#Semantics
> 
> I've heard many upstream developers who were initially very much against
> purging the timestamp when the build was done from their build artifacts
> because they valued the information of when a build was done (whatever their
> reasons are). So this information could simply be retained in that field in 
> the
> .buildinfo file.

I've always claimed that myself, and that was one of the reasons I was
reluctant to eliminate the date from the ar containers, I guess at the
time I could not fully express concretely my gut feeling, but now I
can. :)

The build date is important, because there are actions and events that
are time-based, but are still external to the confinement of the build
environment.

Say, a disk failure corrupting data on the chroot; a broken
debootstrap creating disfunctional chroots, etc, etc. Some of those
might not be immediately visible inside the affected system. But once
known it is useful to be able to say which packages might be suspect
by matching the event date ranges. Of course if the builds end up not
matching other reproducible artifacts then those will be suspect, but
if all reproducers have built using the same external event generator
then that might be harder to see. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Moving towards a deb-buildinfo(5) Format 1.0

[Guillem Jover]

Hi!

As I've mentioned elsewhere, I've noticed several things with the
current .buildinfo format, even after the cleanup pre-merge, that
I'd like to fix or change so that we can hopefully reach Format 1.0.

Some of the issues, that bother me:

* .buildinfo files are not currently signed

  I just need to do that, but given that this is not final I didn't
  see it as a big problem when I noticed just after the first release
  introducing this. Ximin also filed #843925 about this later on.

* .buildinfo filename

  Having the buildinfo filename be derived from its contents and the
  current date is annoying and makes finding it more difficult, as you
  have to scan the .changes file if it's around, but cannot link it
  directly otherwise. It also leaves dangling files around when building
  the same package multiple times, just because the date increases. If
  we have already established that the .buildinfo contents will vary
  across different builders anyway, which means the .changes files
  contents will too, I don't see why we need to distinguish the
  .buildinfo filename generation from the .changes filename.

  So I'd probably prefer to use the same algo as used for .changes
  filename generation. And move the build date inside the .buildinfo
  file in a Build-Date field or similar (as I had for some time on my
  cleanup branch).

* dpkg-genbuildinfo injects itself into debian/files

  I still tend to think this is the right way to do it (instead of
  letting dpkg-buildpackage add the entry). But this can cause issues
  when running it multiple times, but it does not cause them (or
  should not), with a stable filename, so fixing the above would cover
  this one too.

* .buildinfo files are not generated when creating source-only uploads

  I should probably check the exact conditions, but it might be worth
  including a .buildinfo file if it exists and is in debian/files, even
  when doing something like:

$ dpkg-buildpackage -us -uc --change-option=-S

  because then we could "be sure" the maintainer built something. :)

* .buildinfo has some issues when including .dsc information

  Only the .dsc file is referenced not all of its contents, it might
  be better to match .changes logic here. Also the “source”
  pseudo-architecture does not get added to the Architecture field.
  I'll just do at least the latter, I'm open for discussion on the
  former.

* Some of the environment variables seem superfluous or leaks

  Several of the envvars are either always set by dpkg-buildpackage
  anyway (like SOURCE_DATE_EPOCH, although the user might have set it
  explicitly and that will be honored), or should be irrelevant for
  the reproducible build, as we should be fixing packages to make them
  impervious to change in their values (LANG, LC_* etc). Some of which
  kind of leak information about the build system.

  I'm not extremely fussed about this point though, because this might
  still be interesting information to record, but only as long as it
  does not leak possibly sensitive build system information. Locales
  used might be too revealing perhaps. But then reportbug and similar
  leak this and much more so, dunno.


I'll probably do some of the fixes already and bump Format to 0.2
and after the discussion settles we can perhaps do a 0.3, and see how
it goes, and iterate until it looks good, at which point we'd declare
it 1.0, ideally before the freeze. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Guillem Jover]

Hi!

On Wed, 2016-11-09 at 11:16:09 +, Ian Jackson wrote:
> Sven Joachim writes ("Re: misleading timestamps in binnmus"):
> > I'm afraid I don't really have a good suggestion.  Using current date
> > would work but obviously break reproducibility, and any other date seems
> > arbitrary.
> 
> I don't understand why using the current date would break
> reproducibility.
> 
> How does one reproduce a binnmu ?

As Mattia replied, by taking the data from the .buildinfo file:

> Clearly one needs a copy of the binnmu changelog stanza, because that
> (a) determines the version numbers of the generated .debs and (b) is
> actually included in the generated .debs.

man deb-buildinfo, Binary-Only-Changes field. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: misleading timestamps in binnmus

[Guillem Jover]

Hi!

On Tue, 2016-11-08 at 22:41:09 +, Ian Jackson wrote:
> I see the python2.7 source package does this:
> 
>  LAST_CHANGE := $(shell dpkg-parsechangelog -S Date)
>  export BUILD_DATE := $(shell LC_ALL=C date -u +'%b %e %Y' -d 
> '$(LAST_CHANGE)')
>  export BUILD_TIME := $(shell LC_ALL=C date -u +'%H:%M:%S' -d 
> '$(LAST_CHANGE)')
> 
> Is this a recommended recipe ?

I think this should be fine. There's also SOURCE_DATE_EPOCH, that
dpkg-buildpackage honors and otherwise sets now, which can be also
retrieved with «dpkg-parsechangelog -STimestamp», but that should not
be needed here anyway, because…

> AIUI a buildd doing a binnmu will not
> modify the debian/changelog file.

… the changelogs do get a new entry, otherwise everything would fall
apart. Something like this:

,---
zlib (1:1.2.8.dfsg-2+b3) sid; urgency=low, binary-only=yes

  * Binary-only non-maintainer upload for amd64; no source changes.
  * Recompile static libraries with PIE

 -- amd64 / i386 Build Daemon (x86-grnet-01) 
  Sat, 16 Aug 2014 15:12:11 +0100
`---

But I just noticed that the timestamp in this entry seemed suspect,
and it indeed is, as the last maintainer upload was this:

,---
zlib (1:1.2.8.dfsg-2) unstable; urgency=low

  * Drop zlib-bin package as minizip has now been packaged separately,
delay due to lack of notice regarding upload (closes: #753070).

 -- Mark Brown   Sat, 16 Aug 2014 15:12:11 +0100
`---

> The result is that the file timestamps will be lies after a binnmu.  I
> think this is quite undesirable.  Less careful backup programs than
> mine wouldn't notice.  The results would then be a corrupted backup,
> which would break when restored.  There are probably other bad
> consequences.

So the actual problem is that the last timestamp gets reused for the
binNMUs, which seems totally bogus to me. This needs to be fixed in
whatever is injecting the binNMU entries on the buildds.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Repeated entries in Installed-Build-Depends .buildinfo section

[Guillem Jover]

Hi!

On Mon, 2016-10-17 at 13:39:20 +0100, Chris Lamb wrote:
> Excuse terseness; head is elsewhere and I don't want to lose this
> bug report and — alas — I can't submit against src:dpkg yet :)

No problem, and thanks!

>  $ curl 
> https://tests.reproducible-builds.org/debian/buildinfo/unstable/amd64/redis_3.2.4-2_amd64.buildinfo
>  | grep '^ perl '
> 
>  perl (= 5.24.1~rc3-3),
>  perl (= 5.24.1~rc3-3),
> 
> ie. repeated entries in Installed-Build-Depends.

Right, this seems due to a missing dependency simplification step.
I've pushed an untested patch now:

  


will try to test tomorrow, but if some of you can do that, that'd be
much appreciated. ;)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] New default -fdebug-prefix-map build flag for dpkg

[Guillem Jover]

Hi!

On Thu, 2016-07-21 at 11:06:32 +, Mattia Rizzolo wrote:
> On Mon, Jul 18, 2016 at 08:57:01AM +, Mattia Rizzolo wrote:
> > As part of the Reproducible Builds effort [0], we would like to enable
> > a new default build flag from the reproducible/fixdebugpath feature
> > area in order to prevent issues with build paths.

> > Thus, following the dpkg team's policy [5] about adding a new default
> > build flag, I'm seeking a wider discussion to see whether somebody has
> > any concern we haven't already taken care of.
> 
> I think the failing packages are nothing particularly interesting, and
> it's very easy to fix, so if no concerns arise in the next very few
> days, I'm going to open a bug to dpkg to have the build flag turned on
> by default.

This change got included in dpkg 1.18.10.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#787980: status: normalize file permissions when creating control.tar ?

[Guillem Jover]

Hi!

On Mon, 2016-07-04 at 12:17:32 +, Holger Levsen wrote:
> could you please comment briefly on
> your take on this bug and it's status?

I've had my qualms about the need for this patch, but in any
case the provided patch has not been correct now for a while as
I pointed out on IRC some time ago. Which is why Mattia reworked
it temporarily so that the dpkg in the reproducible repo works and
does not mess with the data.tar files.

As I also mentioned on IRC, I'm planning on coming up with something
for this for the next upload. And left it out as it's certainly not
in the critical path to reproducible binaries, as the control member
has a more controlled environment.

> It's in the BTS since 13 months without a maintainer comment.

So it's certainly true that the bug report has had no comment for
that long, and I should have probably updated it for the benefit
of others, but I was actually quite surprised by this mail from you
given that I've been keeping at least Mattia and others on the IRC
channel informed of the progress and bugs status, which I had the
impression were getting relayed to the reproducible IRC channel,
and which he confirms he has been doing all along… so claiming no
maintainer comment seems a bit unfair TBH.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] getting dpkg ready for reproducible sid

[Guillem Jover]

Hi!

On Tue, 2016-05-10 at 10:34:08 +0200, Guillem Jover wrote:
> On Mon, 2016-05-09 at 12:26:13 +, Holger Levsen wrote:
> > b.) common build timestamp for all files created at a later time
> 
> Pending release of new upstream tar. And the patch reworked to stop
> trying to detect the presence of --clamp-mtime at run-time.

Ok, so tar 1.29 has just been released and uploaded to Debian. I've
reworked the patch as mentioned above, and will merge it for 1.18.8,
and will do the same for most of the others.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] getting dpkg ready for reproducible sid

[Guillem Jover]

On Tue, 2016-05-10 at 12:54:04 +0300, Niko Tyni wrote:
> On Tue, May 10, 2016 at 10:34:08AM +0200, Guillem Jover wrote:
> > On Mon, 2016-05-09 at 12:26:13 +, Holger Levsen wrote:
> > > f.) add support for .buildinfo files
> > 
> > I need to finish commenting on this, hopefully in the coming days.
> > 
> > And wasn't it missing at least man pages for the .buildinfo file?
> 
> Thanks, deb-buildinfo(5) is indeed mentioned in #138409 but doesn't
> seem to be done yet. Lunar said in January he'd "rather make sure we all
> agree on the fields before getting to it."

Right, I saw that, but the bulk of the work is writing the man page,
renaming few fields if necessary does not seem comparable, and I'm
fine doing that myself if it ends up being necessary.

> I suppose that mostly refers
> to the ones currently called Environment and Installed-Build-Depends?
> Is consensus any nearer now than in February?

Sorry, I've not gone over the thread again yet.

> > > g.) teach dpkg-genbuildinfo the new --build=[...] syntax
> > 
> > This should be squashed in the existing patch before submission.
> 
> It's squashed in the current pu/reproducible_builds branch fwiw,
> this was just a changelog comment.

Ah ok, perfect!

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] getting dpkg ready for reproducible sid

[Guillem Jover]

Hi!

On Mon, 2016-05-09 at 12:26:13 +, Holger Levsen wrote:
> Looking at this I notice we have some changes without bugs and some
> changes involving several bugs, still I think it would be good to
> discuss them as uploaded.
> 
> IOW: dear dpkg maintainers, what are your comments regarding getting the
> following changes into sid:
> 
> a.) single timestamp for ar headers

Pending rereview of local changes, and staged until the treewalk
changes have brewed in the rebootstrap, reproducible rebuilds and
in unstable for a bit.

I might have added this earlier if the treewalk code had been tested
earlier, but didn't want to entangle these kind of changes together
w/o prior wide testing.

> b.) common build timestamp for all files created at a later time

Pending release of new upstream tar. And the patch reworked to stop
trying to detect the presence of --clamp-mtime at run-time.

> c.) build timestamp using SOURCE_DATE_EPOCH

Should be fine, I've added infrastructure to not have to use
date(1), but otherwise should get in probably for 1.18.8 already.

> d.) preset build timestamp to latest changelog entry

I've been reluctant to consider this before the .buildinfo is
generated and ideally actually stored in ftp-master. But I think
at this point, I think I'll just merge it, so that we can get
reproducible ar containers in the archive. Or I could merge WIP
.buildinfo support and mark it as Format 0.0 and then we can tune
it until we consider it ready and then mark it 1.0, let's see.

> e.) normalize file permissions when creating control.tar

The current change in the repro git is not correct as it changes the
perms for control.tar and data.tar. The side-effects of refactoring!
I've to take a look at this.

> f.) add support for .buildinfo files

I need to finish commenting on this, hopefully in the coming days.

And wasn't it missing at least man pages for the .buildinfo file?

> g.) teach dpkg-genbuildinfo the new --build=[...] syntax

This should be squashed in the existing patch before submission.

> And, do we need bugs for a+b+c+g ?

No need.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#819194: dpkg-buildflags: please add normalizedebugpath to reproducible feature set

[Guillem Jover]

Hi!

On Tue, 2016-03-29 at 21:52:53 -0400, Holger Levsen wrote:
> On Tue, Mar 29, 2016 at 09:36:00PM -0400, Daniel Kahn Gillmor wrote:
> > This isn't fun-spoiling, it's a useful reality check.  But if we were
> > required to get all the way to 100% before we made any progress, then
> > reproducible builds wouldn't have gotten off the ground at all.
> 
> it's surely progress on the gcc/clang side of things but dropping the
> build path from the .buildinfo files would be a huge step *backwards*
> for reproducibility…

I concur with Daniel, I don't see anyone suggesting to drop the build
path from the .buildinfo file just yet. But as long as that field does
not leak information I'm ok with having it there. I think eventually
it should be dropped because nothing should be recording the build
path in the build, and the build should really be independent of that.

> > The changes proposed in this bug report are a good step that should
> > handle a very large proportion of the debian archive.  The fact that
> > there will remain corners of the archive that need additional work is
> > fine: we can turn our attention to the remaining 20% once we get 80% of
> > the buildpaths resolved.
> 
> true.
> 
> my point was: I think we still need the build path in the .buildinfo files.

For now probably yes.

> Also, c/c++ packages today only make up a small portion of the archive.
> Probably this famous someone should do a rebuild of the archive, using
> our toolchain (and this patch), using arbitrary build pathes.

That's why I mentioned that the other language flages be included so
that we get better coverage besides C/C++.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#818414: diffoscope: Please add full support for deb(5)

[Guillem Jover]

On Wed, 2016-03-16 at 23:10:13 +0100, Guillem Jover wrote:
> Package: diffoscope
> Version: 51
> Severity: wishlist

> It seems diffoscope does not fully support the deb(5) format as
> specified in the man page. At least control.tar.xz members, and
> probably neither control.tar nor data.tar.
> 
> It would be nice if these could be supported, as I might try to
> get these accepted by dak.

It was mentioned off-bts that this was not entirely clear, so let
me fix that.

The program was added to <https://wiki.debian.org/Teams/Dpkg/DebSupport>
as not supporting the aforementioned .deb ar members, which is
implemented in diffoscope/comparators/deb.py. The full .deb format
is specified in the deb(5) man page, and anything there should ideally
be supported by 3rd-party parsers, as otherwise they might fail to
process binaries that can be generated now or have been in the past by
dpkg-deb or other tools.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#818414: diffoscope: Please add full support for deb(5)

[Guillem Jover]

Package: diffoscope
Version: 51
Severity: wishlist

Hi!

It seems diffoscope does not fully support the deb(5) format as
specified in the man page. At least control.tar.xz members, and
probably neither control.tar nor data.tar.

It would be nice if these could be supported, as I might try to
get these accepted by dak.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] [Reproducible] On making Stretch self-contained IRT to reproducibility

[Guillem Jover]

Hi!

On Wed, 2016-03-09 at 10:32:08 +0100, Holger Levsen wrote:
> On Mittwoch, 24. Februar 2016, Emilio Pozuelo Monfort wrote:
> > On 24/02/16 22:16, Niels Thykier wrote:
> > >- Possible lack of buildd resources to do the rebuild.  Notably, due
> > >  to Multi-Arch:same we would generally need to do the rebuild on all
> > >  architectures.
> > FWIW, that would be solved with #758616.
> 
> this bug doesnt look like it will be fixed anytime soon. (Also I'm not 
> convinced the bug this one is blocked by (#681289: "changelog and copyright 
> should be metadata") will really be implemented as suggested.)

Actually I've got code implementing the change in principle, I just
need to write the functional test cases. But recently I realized that
it might potentially get in the way of fixing a bug I found in the
current refcounting. :/ So I'll need to check that first. I can take
some time to publish more details if there's interest.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] symlink permission bits on non-Linux

[Guillem Jover]

[ Just stumbled on this on the mail archive. ]

Hi!

On Tue, 2016-02-16 at 08:42:07 +0100, Jérémy Bobbio wrote:
> Steven Chamberlain:
> > On linux, a symlink can only have permissions 0777 (lrwxrwxrwx)
> > 
> > But on at least kfreebsd (maybe hurd?) there is no such limitation, and
> > permissions are set like any regular file.  That also means the umask is
> > applied...  and tar and dpkg-deb preserve this.

Yes, Hurd too. Linux probably is the odd one here.

> > This proves to be an issue for:
> >   * reproducible builds on kfreebsd, affected by user's umask
> >   * reproducing arch:all packages between linux<->kfreebsd
> >   * reproducing linux packages by cross-building from kfreebsd
> > 
> > I think we should normalise symlinks' permissions to 0777, except GNU
> > chmod can't do that!  (chmod follows the symlink, and has no -h flag).
> > 
> > Adding a -h (no dereference) option to chmod would allow dh_fixperms to
> > use that.  But (as pointed out in #759886) adding things there does not
> > help packages not using debhelper, or other uses of tar.
> > 
> > Would this be best added as a feature to tar, that dpkg-deb can use?
> > Probably a new flag, that would apply --mode a=rwx only to symlinks.
> > 
> > Or are there other ideas how to fix this?

I noticed this due to the dpkg test suite, and fixed it by temporarily
setting umask to 0:

  


And added it to the wiki:

  
.

> One idea floating is to get dpkg-deb working with an explicit manifest
> to create the package content. I believe that would solve the issue. But
> that's at least mid-term because dpkg needs to get its own Tar
> implementation (or maybe depend on libarchive) and, likely harder, a
> format needs to be defined for the manifest.

Right.

> In the meantime, shouldn't GNU chmod get a `-h` option in any cases if
> it's going to be used on kFreeBSD?
> 
> Then it's pretty easy to start with `dh_fixperms` and see how much it
> helps.
> 
> Guillem said he was ok with dpkg depending on recent versions of
> Tar[1], but changes would need to be accepted by Tar upstream.

Yes, if you get a new option in tar I'm fine using it. But given the
reaction to the --clamp-mtime option, this might not fly with upstream?
(Also it seems the patch for that one might have been better sent to
bug-tar instead of help-tar.)

> What's the situation regarding symlinks on HURD?

See above.

>  [1]:?https://bugs.debian.org/759886#73

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

On Thu, 2016-02-04 at 09:44:13 +0100, Guillem Jover wrote:
> (BTW I also realized that I don't think we are including «Essential:yes»
> packages in that set, and we should.)

Actually, disregard this, they are already included! Sorry for the
noise.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Sun, 2016-01-31 at 14:43:08 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > > How about naming the field “Environment-Variables”?
> > 
> > Hmm, or Environment, or Build-Environment, which reminds me that I've
> > found the usage of Build-Environment (as the list of transitively
> > required packages) slightly confusing, precisely because the first
> > thing that comes to mind with environment is the variable space.
> > 
> > Perhaps we should consider renaming that one? Say Build-Packages (but
> > that might be confusing), Build-Depends-Used, or something else? We
> > also already have a Built-Using field too (although for source
> > packages not binary ones, with a name I've also found slightly
> > confusing as being too generic).
> 
> Ok. What about “Environment” for the variables,

I'm not sure if it'd be better to be explicit about this being a build
thing, and not just a random environment. Are you worried about confusion
with the previous usage of the field with the same name?

> and “Installed-Build-Depends” for the list of packages?

I asked for more suggestions on #debian-dpkg, and Johannes Schauer
suggested Transitive-Build-Depends, which is something I had in mind
too (that or «Recursive-»), but kind of softly discarded in trying to
have a consistently namespaced «Build-» field name. :) Some of the
reasons Johannes put forward are that this name is better because it
clearly describes what's the exact purpose of the field, and gives
no room for misinterpretation. And if we had to change the algorithm
we could just use a new name. All of which I concur with.

(BTW I also realized that I don't think we are including «Essential:yes»
packages in that set, and we should.)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Tue, 2016-02-02 at 16:34:40 +0100, Jérémy Bobbio wrote:
> We were discussions the restrictions on buildinfo identifiers:
> 
> fweb_1.62-12+b2_brahms-20120530T114812Z.buildinfo
> ^^^
>   this part
> 
> The proposal was “the string should consist only of alphanumeric
> characters and hyphens”. Guillem made the following comment while
> reviewing the patches for dpkg:
> 
> Guillem Jover:
> > Can we just simply use the package name rules instead? It also avoids
> > potential problems with case and similar. (There's a
> > pkg_name_is_illegal function in Dpkg::Package already.)
> 
> After reaching out to Ansgar with a patch for dak to implement the
> above, he replied:
> 
> Ansgar Burchardt:
> > The allowed sets for package names and the suffix of buildinfo
> > filenames won't be the same even with that change.  However currently
> > the suffix of buildinfo filenames matches what is allowed for .changes
> > files.

Ah ok matching what is allowed in «.changes» with what is allowed in
«.buildinfo» makes sense. But…

> > I'm not sure why allowing capital letters used in the suffix of
> > buildinfo files should be an issue? After all we allow capital letters
> > for both version numbers (part of the filename) and in names of changes
> > files.

…are uppercase letters allowed for the third (arch) component in
.changes files? No architecture should be uppercase anyway.

I'd just like to avoid allowing things that then becomes very
difficult to ban after the fact.

> > (In the other direction not everything allowed as a package name can be
> > used as the suffix of .changes and .buildinfo files either.)

True, which would introduce more undesired things.

> Guillem, any further comments? Do you have any strong opposition to the
> initial proposal?

Actually after consideration, I don't think I have, more so if it
makes representing dates in an incorrect way (the ‘Z’).

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

On Sat, 2016-01-30 at 15:18:30 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > Lunar:
> > > I think the proposed patch is missing a field to record some environment
> > > variables that can affect the build process. Right now, I'm thinking of
> > > DEB_BUILD_OPTIONS and DEB_flag_{SET,STRIP,APPEND,PREPEND} (from
> > > dpkg-buildflags). Maybe others? Build profiles?
> > >
> > > Initially I was against recording such information, but now that we
> > > understand the purpose of .buildinfo files better and not mandate that
> > > they be reproducible themselves, it doesn't matter if one contains
> > > `DEB_BUILD_OPTIONS=parallel=4` and the other
> > > `DEB_BUILD_OPTIONS=parallel=16`. It is the responsibility of users
> > > trying to recreate a given package to filter this out.
> > 
> > Hmm right, makes sense, but I see this also to be a bit problematic.
> > There are many variables that do affect the build which we'd need to
> > record. Including all of the them seems like another privacy
> > concerning issue. Whitelisting, we might end up missing some but it's
> > privacy-safe; blacklisting we might end-up including sensitive ones,
> > but not miss any build-related ones, which is privacy-unsafe.
> > 
> > Some things that come to mind that do affect the build in a significant
> > way: CC, LD_LIBRARY_PATH, DEB*, DPKG_*, PATH, MAKEFLAGS.
> > 
> > The traditional build flags (i.e. CFLAGS, LDFLAGS, etc) might also affect
> > the build depending on the rules file.
> 
> I was thinking of a whitelist approach and to start with use cases we
> can already think of, adding more variables to record if we identify
> them as missing in the future.

That works for me, and it's no worse than our current support for
build flags, where we have a whitelist too. So with new build flags
we need to explicitly add support for them too. Althought here we'd
like to catch build variables even if dpkg-buildflags does not support
them (say CXX).

> How about naming the field “Environment-Variables”?

Hmm, or Environment, or Build-Environment, which reminds me that I've
found the usage of Build-Environment (as the list of transitively
required packages) slightly confusing, precisely because the first
thing that comes to mind with environment is the variable space.

Perhaps we should consider renaming that one? Say Build-Packages (but
that might be confusing), Build-Depends-Used, or something else? We
also already have a Built-Using field too (although for source
packages not binary ones, with a name I've also found slightly
confusing as being too generic).

> I will also add another vendor hook for the list of variables.

I don't think this is necessary in this case, all current variables
seem pretty dpkg-generic to me. Or do you see any that is
Debian-specific?

> > Build profiles are already recorded in the binary packages, but having
> > that in the .buildinfo file seems right as it makes it easier to
> > reproduce the build environment.
> 
> Should it be a new field or would recording the DEB_BUILD_PROFILES
> variable be enough?

These can be passed also through the -P option to dpkg-buildpackage and
dpkg-checkbuilddeps, but this is not currently possible for all scripts
that use build profiles, and that's why dpkg-buildpackage sets the
DEB_BUILD_PROFILES variable from the command-line option value, so I
think just recording the environment is enough for now. We can always
also record them in a Build-Profiles field if need be.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Fri, 2016-01-29 at 16:07:54 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > > One of the main change is that `.buildinfo` should now be named with an
> > > arbitrary identifier. By default this defaults to $HOSTNAME-$TIMESTAMP
> > > but can be set to an arbitrary value by the `--buildinfo-identifier`
> > > command line flag.
> > 
> > Hmmm, leaking the hostname seems slightly privacy-concerning? If the
> > information therein is not relevant I'd rather use something like an
> > UUID (although that would require increasing the pseudo-build-essential
> > set), or just hashing the hostname-timestamp with something like md5
> > or sha1 or similar.
> 
> My hunch is that having a timestamp visible in the file name might help
> recognizing files quickly after a bunch of builds, especially to
> identify the last one. So I'd rather keep it.

I see no problem at all with that, and seems like a good idea. I only
take slight issue with the hostname really.

> If privacy is the goal, hashing just the hostname would not be help
> much, as any precomputed table would work.
> 
> How about $TIMESTAMP-$EIGHT_FIRST_CHARS_OF_BUILDINFO_MD5?
> 
> (I'm picking md5 because it's already used in dpkg-gensymbols.)

Yeah md5 is not security sensitive here, so it's good enough. Using
$timestamp-$selfmd5 seems good. And as long as we specify that this
format could change in the future I think we leave room to maneuver.

> > Can we just simply use the package name rules instead? It also avoids
> > potential problems with case and similar. (There's a
> > pkg_name_is_illegal function in Dpkg::Package already.)
> 
> Sounds reasonable. I've updated the wiki page and prepared a patch for
> dak.

Thanks!

> > > +} else {
> > > +warning(_g('no .dsc file, skipping .buildinfo generation'));
> > > +}
> > >  }
> >
> > ISTR mentioning this before, but I don't see why generating the
> > buildinfo file is tied to existing a source package at all? The source
> > should be included if we are including a source in the upload, that's
> > it.
> 
> The whole puprose of the reproducible builds effort is to provide a
> verifiable path from sources to binaries. Signed .buildinfo files are
> certifications that the listed binary packages have been built using the
> described source and environment.
> 
> Only listing the source in .buildinfo when a source is included in the
> upload would prevent us to have multiple builders certify the same
> binaries. That would cut us from providing multiple certifications and
> would undermine the purpose of reproducible builds.
> 
> So I could remove the limitation, but the resulting .buildinfo file
> would not be very useful for reproducible builds.

I understand why you need the chain, but don't forget that this quite old
bug report was about including build information in uploads, which seems
to me is a superset of the needs of the reproducibility project! So I've
to take into account the general case here too. :)

Also I kind of fail to see a problem here. With the current implementation
when there is no source, the .buildinfo file is not generated at all. It
seems to me switching to generating it even when there is no source is no
worse (it's actually better!) than not including a .buildinfo at all.

> > > +$fields->{'Source'} = $spackage;
> > > +if ($changelog->{'Binary-Only'}) {
> > > +$fields->{'Source'} .= ' (' . $sourceversion . ')';
> > > +$fields->{'Changes'} = $changelog->{'Changes'} . "\n\n"
> > > + . ' -- ' . $changelog->{'Maintainer'}
> > > + . '  ' . $changelog->{'Date'};
> > > +}
> > 
> > Hmmm, it bothers me slightly that the Changes field here diverges in
> > form from the one in the .changes file.
> 
> I can understand. It's been designed that way because it's actually only
> there for binNMUs where the source is the same as the original and we
> need a way to reconstruct the right changelog file.
> 
> Because sbuild might actually change its strings in the future, it felt
> like plain copy/pasting was the safest. So recreating the changelog in
> case of binNMU is about outputing the value of the Changes field in the
> .buildinfo, a blank line, and the changelog from the original source.

Sure.

> > I think I'd prefer to have the Date as its own field, maybe always
> > included. And also the Maintainer field. Any reason to not include
> > t

Re: [Reproducible-builds] Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Thu, 2016-01-28 at 19:36:25 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > I've some pending changes I'll be committing to master or a separate
> > branch, that I'd like to be tested on the reproducible setup (ideally
> > against the already generated and pre-existing reproducible binaries),
> > if that's possible, I'll ask about that when those land, I just need
> > to finish up fewm more unit tests.
> > 
> > Here's a quick review: […]

> I can spend the next days improving the patch with your remarks, but I'd
> rather not duplicate your work. :)

BTW make sure «DPKG_DEVEL_MODE=1 make check» passes (in case you were
not doing so before :), which should help to trap some problems with
the perl code.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

zOn Thu, 2016-01-28 at 19:01:59 +0100, Holger Levsen wrote:
> On Donnerstag, 28. Januar 2016, Guillem Jover wrote:
> > Hmmm, leaking the hostname seems slightly privacy-concerning? If the
> > information therein is not relevant I'd rather use something like an
> > UUID (although that would require increasing the pseudo-build-essential
> > set), or just hashing the hostname-timestamp with something like md5
> > or sha1 or similar.
> 
> yeah, "something / anything" is fine. dak / the archive software can rename 
> it 
> anyway, as it likes.

Ah ok, perfect then.

> (I'd be in favor of naming the first accepted buildinfo 
> file of the archive just "" so that it's predictable…

I'm not sure how we'd use a sequential number in a distributed manner
starting with 0s though? :9

> > I've some pending changes I'll be committing to master or a separate
> > branch, that I'd like to be tested on the reproducible setup (ideally
> > against the already generated and pre-existing reproducible binaries),
> > if that's possible, I'll ask about that when those land, I just need
> > to finish up fewm more unit tests.
> 
> That's possible, though not (yet nor in near future) against pre-existing 
> binaries. (We lack the code for that.)
> 
> What we can do easily, is build and upload dpkg to our repo and use it to 
> build the whole Debian archive on amd64, which roughly takes 8 days for both 
> sid+stretch, and thus roughly 4 days for one suite, if we disable building 
> the 
> other. (Which we can definitly do, especially if we don't disable building of 
> new versions in that other suite…)

Oh ok, that's a bit unfortunate, it would be nice to be able to catch
any possible regressions in the generated binaries, and it would also
show how to use reproducible builds to see that nothing has actually
changed, even after implementation changes in the toolchain.

On Thu, 2016-01-28 at 19:02:53 +0100, Holger Levsen wrote:
> +many thanks for your thorough review! :-)

No problem!

Regards,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Thu, 2016-01-28 at 19:36:25 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > I've some pending changes I'll be committing to master or a separate
> > branch, that I'd like to be tested on the reproducible setup (ideally
> > against the already generated and pre-existing reproducible binaries),
> > if that's possible, I'll ask about that when those land, I just need
> > to finish up fewm more unit tests.
> > 
> > Here's a quick review: […]
> 
> Thanks for the review! Just so I can organize my work, are your pending
> changes related to the .buildinfo?

Nope, they are for the sorted file lists, and ar member stuff.

> I can spend the next days improving the patch with your remarks, but I'd
> rather not duplicate your work. :)

Sure, go ahead. :)

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Wed, 2016-01-27 at 08:58:47 +0100, Jérémy Bobbio wrote:
> Jérémy Bobbio:
> > The attached patch will enable dpkg-buildpackage to create .buildinfo
> > files as specified on the Debian wiki [1]. They have two main purposes:
> > 
> >  * recording information about the system environment used during a
> >particular build—versions of the build dependencies installed, system
> >architecture, etc. for easier forensics/debugging;
> >  * describe how to recreate (partially or in full) the original
> >environment when trying to reproduce a particular build.
> 
> I think the proposed patch is missing a field to record some environment
> variables that can affect the build process. Right now, I'm thinking of
> DEB_BUILD_OPTIONS and DEB_flag_{SET,STRIP,APPEND,PREPEND} (from
> dpkg-buildflags). Maybe others? Build profiles?
>
> Initially I was against recording such information, but now that we
> understand the purpose of .buildinfo files better and not mandate that
> they be reproducible themselves, it doesn't matter if one contains
> `DEB_BUILD_OPTIONS=parallel=4` and the other
> `DEB_BUILD_OPTIONS=parallel=16`. It is the responsibility of users
> trying to recreate a given package to filter this out.

Hmm right, makes sense, but I see this also to be a bit problematic.
There are many variables that do affect the build which we'd need to
record. Including all of the them seems like another privacy
concerning issue. Whitelisting, we might end up missing some but it's
privacy-safe; blacklisting we might end-up including sensitive ones,
but not miss any build-related ones, which is privacy-unsafe.

Some things that come to mind that do affect the build in a significant
way: CC, LD_LIBRARY_PATH, DEB*, DPKG_*, PATH, MAKEFLAGS.

The traditional build flags (i.e. CFLAGS, LDFLAGS, etc) might also affect
the build depending on the rules file.

Build profiles are already recorded in the binary packages, but having
that in the .buildinfo file seems right as it makes it easier to
reproduce the build environment. Ideally, parallel=N should not have
any visible effect but I guess it currently might. Most of the other
DEB_BUILD_OPTIONS do have a visible effect on the artifacts generated.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#138409: dpkg-dev: please add support for .buildinfo files

[Guillem Jover]

Hi!

On Tue, 2016-01-05 at 14:32:51 +0100, Jérémy Bobbio wrote:
> Control: retitle -1 dpkg-dev: please add support for .buildinfo files
> Control: tag -1 + patch

> The attached patch will enable dpkg-buildpackage to create .buildinfo
> files as specified on the Debian wiki [1]. They have two main purposes:
> 
>  * recording information about the system environment used during a
>particular build—versions of the build dependencies installed, system
>architecture, etc. for easier forensics/debugging;
>  * describe how to recreate (partially or in full) the original
>environment when trying to reproduce a particular build.
> 
> Since Guillem's preliminary review in February 2015 [2], the
> specification has slightly elvolved to be a bit more relaxed and the
> code have been improved.

Cool, thanks!

> One of the main change is that `.buildinfo` should now be named with an
> arbitrary identifier. By default this defaults to $HOSTNAME-$TIMESTAMP
> but can be set to an arbitrary value by the `--buildinfo-identifier`
> command line flag.

Hmmm, leaking the hostname seems slightly privacy-concerning? If the
information therein is not relevant I'd rather use something like an
UUID (although that would require increasing the pseudo-build-essential
set), or just hashing the hostname-timestamp with something like md5
or sha1 or similar.

> To address privacy concerns, the Build-Path field is now only included
> when either the build path starts by `/build/` or
> `--always-include-path` has been specified on the command line of
> `dpkg-genbuildinfo`.

Thanks!

> .buildinfo files are now accepted (although discarded) by the Debian
> archive [3]. This change should thus not affect Debian developpers in
> their daily work.

Ah perfect, thanks for pushing for this. I'm planning on including
most of the patches that look fine for 1.18.5 or .6 ideally.

I've some pending changes I'll be committing to master or a separate
branch, that I'd like to be tested on the reproducible setup (ideally
against the already generated and pre-existing reproducible binaries),
if that's possible, I'll ask about that when those land, I just need
to finish up fewm more unit tests.

Here's a quick review:

> From 7b6d953f834b1e8800d3f8af4570d57d86e5c592 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Bobbio?= 
> Date: Fri, 6 Nov 2015 12:17:39 +
> Subject: [PATCH] Add support for .buildinfo files

> diff --git a/man/dpkg-buildpackage.1 b/man/dpkg-buildpackage.1
> index 13770ba..54cee7b 100644
> --- a/man/dpkg-buildpackage.1
> +++ b/man/dpkg-buildpackage.1
> @@ -317,6 +322,12 @@ The source package version (without the epoch).
>  The upstream version.
>  .RE
>  .TP
> +.BI \-\-buildinfo-identifier= identifier
> +By default, \fBdpkg\-buildpackage\fP put the system hostname and the

“puts”, but perhaps better “uses foo and bar to create the quux
filename”.

> +current time in the name of the \fB.buildinfo\fP file. An arbitrary
> +identifier can be specified as a replacement (since dpkg 1.18.5).

I'd probably describe first what the option actually does, so that the
version when it got intruduced makes sense to what it refers to. And
then how does that diverges from the default.

And please, place new sentences on a new line. (I should probably
update the "coding-style" about that.)

> It must
> +contain only alphanumeric characters and hyphens.
> +.TP
>  .BI \-p sign-command
>  When \fBdpkg\-buildpackage\fP needs to execute GPG to sign a source
>  control (\fB.dsc\fP) file or a \fB.changes\fP file it will run

> diff --git a/man/dpkg-genbuildinfo.1 b/man/dpkg-genbuildinfo.1
> new file mode 100644
> index 000..77f2a76
> --- /dev/null
> +++ b/man/dpkg-genbuildinfo.1
> @@ -0,0 +1,98 @@

> +.BI \-\-always\-include\-path
> +By default, the \fBBuild-Path\fR field will only be written if the current
> +directory starts with \fB/build/\fR. Specify this option to always write
> +a \fBBuild-Path\fR field when generating the \fB.buildinfo\fR.

Missing escaped dash in field names.

> diff --git a/scripts/Dpkg/Control/Types.pm b/scripts/Dpkg/Control/Types.pm
> index 09e12d1..ad6e11b 100644
> --- a/scripts/Dpkg/Control/Types.pm
> +++ b/scripts/Dpkg/Control/Types.pm
> @@ -51,16 +52,17 @@ between Dpkg::Control and Dpkg::Control::Fields.
>  
>  use constant {
>  CTRL_UNKNOWN => 0,
> -CTRL_INFO_SRC => 1,  # First control block in debian/control
> -CTRL_INFO_PKG => 2,  # Subsequent control blocks in debian/control
> -CTRL_INDEX_SRC => 4, # Entry in repository's Packages files
> -CTRL_INDEX_PKG => 8, # Entry in repository's Sources files
> -CTRL_PKG_SRC => 16,  # .dsc file of source package
> -CTRL_PKG_DEB => 32,  # DEBIAN/control in binary packages
> -CTRL_FILE_CHANGES => 64, # .changes file
> -CTRL_FILE_VENDOR => 128, # File in $Dpkg::CONFDIR/origins
> -CTRL_FILE_STATUS => 256, # $Dpkg::ADMINDIR/status
> -CTRL_CHANGELOG => 512,   # Output of dpkg-parsechangelog
> +   

Re: [Reproducible-builds] Bug#759999: dpkg-dev: please make mtimes of packaged files deterministic

[Guillem Jover]

Hi!

On Fri, 2016-01-15 at 22:28:12 +0100, Jérémy Bobbio wrote:
> reassign 759886 dpkg-dev
> retitle 759886 dpkg-dev: please make mtimes of packaged files deterministic
> thanks

Thanks for the patches!

> The second patch will use the --mtime and --clamp-mtime option of tar to
> clamp the mtime of files recorded in control.tar and data.tar to the
> build timestamp: files created at a later time will see their mtime
> set to the build timestamp (that's #759886). As --clamp-mtime is only
> available since tar/1.28-1 and has not yet been merged upstream,
> dpkg-deb will first look for its availability by looking for the option
> in the output of “tar --help”. If it's not available, it will fallback
> to the previous behavior.

The --clamp-mtime options looks primising, but I'm not happy about its
current situation. Please get this upstreamed first. I don't mind making
dpkg depend on an extremely recent GNU tar, but I really don't want to
use a Debian specific option like this, as it could change semantics,
etc. And I don't like parsing --help output at all. I still have to fix
the situation with dpkg-source using the Debian specifi gzip --rsyncable
option, for example. :(

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] dpkg_1.18.1.0~reproducible5 ftbfs

[Guillem Jover]

On Tue, 2015-08-04 at 08:42:48 -0700, Vagrant Cascadian wrote:
> On 2015-07-31, Guillem Jover wrote:
> > Right, I noticed this quite some time ago, but forgot to bring it up.
> > W/o having checked anything, it might be that whoever prepared the
> > release perhaps forgot to «autoreconf -f -i» the sources and prepared
> > it from a previous .dsc instead of a git tree?
> 
> I *think* what's happening is the reproducible builds git tree doesn't
> include tags for the version, and .dist-version isn't present, so
> ./get-version is returning an empty string (it can't get it from git or
> From .dist-version), so "autoreconf -f -i" fails... running:
> 
>   echo 1.18.1.0~reproducible5 > .dist-version
>   autoreconf -f -i
>   debuild -us -uc # or your build of choice
> 
> Seems to fix/workaround the issue.

Ah, the problem is that the releases are being made directly from
git with dpkg-buildpackage. The correct way to release dpkg is to
«make dist» and then build Debian packages from that tarball.

The detailed instructions on releasing can be found on the “How to
release“ section here:

  <https://wiki.debian.org/Teams/Dpkg/GitUsage>

although several points might not be relevant here.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] dpkg_1.18.1.0~reproducible5 ftbfs

[Guillem Jover]

Hi!

On Fri, 2015-07-31 at 16:49:13 +0200, Holger Levsen wrote:
> so yesterday I tried to build 
> http://reproducible.alioth.debian.org/debian/dpkg_1.18.1.0~reproducible5.dsc 
> with pbuilder on sid/armhf and that failed _exactly_ like 
> https://reproducible.debian.net/rbuild/unstable/amd64/dpkg_1.18.1.0~reproducible5.rbuild.log
> 
> I then tried to build dpkg_1.18.1.dsc from sid proper and that built 
> flawlessly on sid/armhf.
> 
> Thus I conclude our reproducible patches cause this, whatever this is. Yet I 
> lack time atm to debug this, so this mail is merely a note to Guillem and a 
> call for help to the reproducible folks.

Right, I noticed this quite some time ago, but forgot to bring it up.
W/o having checked anything, it might be that whoever prepared the
release perhaps forgot to «autoreconf -f -i» the sources and prepared
it from a previous .dsc instead of a git tree?

> I've now also added dpkg to the 
> https://reproducible.debian.net/issues/unstable/ftbfs_in_jenkins_setup_issue.html
>  
> issues, which will make sure the ftbfs issue will soon (tomorrow, I think) 
> not 
> show up on https://tracker.debian.org/pkg/dpkg anymore.

Appreciated.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Storing .deb checksums in ADMINDIR/status?

[Guillem Jover]

Hi!

On Tue, 2015-06-23 at 09:31:05 +0200, Jérémy Bobbio wrote:
> Some people suggested that we should record a checksum of the `.deb`
> installed as a way to unambiguously referring to a specific package.

In principle the tuple pkgname-version-arch should be unique per
archive, otherwise bad-things-will-happen. Of course that does not
cover locally built packages and similar, or mixing different archives
with duplicated tuples, but then those are probably out-of-scope for
reproducible builds *in* Debian anyway, I guess.

> The main benefit that I can think of is that it would allow to directly
> retrieve the file from snapshot.debian.org based on the hash‗[2].

Personally I find the point that David mentioned to be a bit more
interesting. :)

> But, as far as I know, this information is currently not recorded by
> dpkg and there is no way to know for sure which `.deb` has been used for
> a package currently installed. I have a couple of memories where this
> could have been useful outside of the aforementioned use case.
>·
> From my limited knowledge of dpkg's internals, computing checksums
> and adding a new field to the status file doesn't seem hard to
> implement.

The general idea seems worthwhile in principle. The devil is in the
details though, and with dpkg, the implementation is usually not the
hard part. :)

David also pointed some of the possible issues. Others that quickly
come to mind, would be:

 * Checksum of what exactly? Although the seemingly obvious answer
   might be “the entire .deb container”, depending on what one wants,
   the interesting data might be different. For example, essential for
   apt would appear to be control.tar and data.tar, and you might not
   want to reinstall if some other member changes; when using signed
   packages changes to the signatures might also be relevant. Other
   .deb members might also be relevant in case another tool wants to
   use them.
 * Currently dpkg extracts the control.tar with dpkg-deb directly to
   disk, and gets the data.tar contents piped from dpkg-deb, so it does
   not get direct access to the whole file, which means the checksum
   would need to be computed out-of-band, needing to process the .deb
   one more time, which might be wasteful.
 * A possibility could be to pre-compute the checksum on creation or
   modification time, and store it in the debian-binary member for
   example. The problem with that is that tools that modify .debs
   might not genereate a checksum, or worse might not update it. And
   this would also not benefit old binaries.
 * Another possibility might be to make dpkg-deb compute the checksum
   when parsing the .deb and output it on a supplied fd through a
   command-line option.
 * Even when dpkg was being used through dselect, where the checksums
   from the archive were fresh and at reach from the available file,
   dpkg has never propagated them to the status file. I guess mainly
   because at the time of «dpkg -i», there was no guarantee that those
   packages corresponded to the ones from the archive.
 …

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Heads up: Upcoming dpkg-buildpackage -j precedence change

[Guillem Jover]

[ reproducible-builds people, please see below. ]

Hi!

On Tue, 2015-05-12 at 10:02:27 +0100, Jonathan Dowland wrote:
> On Mon, May 11, 2015 at 08:40:16PM +0200, Guillem Jover wrote:
> > $ make -jN -f debian/rules build
> > 
> > and
> > 
> > $ DEB_BUILD_OPTIONS=parallel=N debian/rules build
> 
> I prefer the latter behaviour but the former brevity. Would you consider
> something like -J in dpkg-buildpackage adjusting parallel= in
> DEB_BUILD_OPTIONS, and perhaps in the future phasing out -j?

Sure, adding either an option to disable the forced parallel runs in
combination with -j, or a new option such as -J seems fine to me.

The “nice” thing about -j is that it allows anyone to naturally exercise
parallel builds (like they'd do with an upstream project) and possibly
notice if something does not support them. Getting rid of it would mean
that the current marking of packages might be even slower than it is.
Because we have to opt-in for every and each package in the archive,
with the usual multi-year Debian adoption rates and probably longer in
this case. So I think having both semantics at hand is useful.

And of course «dpkg-buildpackage -jN» does not cover build systems that
do not use make, but I've always thought of DEB_BUILD_OPTIONS=paralle=N
as a way to request that.

> > Actually Makefiles that do not support parallel builds and do not
> > declare so with .NOTPARALLEL: are buggy, because upstreams do not
> > have any standardized opt-in way to honor a parallel build request.
> 
> Now that rules files have to be makefiles; I wonder if this should be 
> clarified
> in policy, one way or the other. That is: should debian/rules be assumed to
> support parallel building by default, or not? I imagine the saner default for
> us would be not (even though I am a big fan of parallel builds).

IMO dpkg-buildpackage should assume yes, in the same way it assumes
packages are cross-buildable, and we don't go around marking them as
such. But I guess for Debian that depends on how much of our packaging
and upstream build systems are parallel buildable. I'd have assumed that
with projects like Gentoo around and multicore systems being so common
nowadays, many things would be parallel buildable already, although I
don't have numbers…

… but, we do have a way to check if a package is parallel buildable now,
through reproducible builds. If a package outputs exactly the same with
dpkg-buildpackage -j1 and -jauto (w/o DEB_BUILD_OPTIONS) then it means
the package is good, otherwise it might need fixing one way or another.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo

[Guillem Jover]

Hi,

On Wed, 2015-02-04 at 14:36:12 +0100, Holger Levsen wrote:
> On Sonntag, 1. Februar 2015, Guillem Jover wrote:
> >  * I'm still somewhat unconvinced that having byte-for-byte identical
> >container binary .deb packages is the ideal minimal reproducible
> >unit.
> 
> I'm getting more and more convinced it is. Cause that is what we really care 
> about.

Different people care about very different things, in very different
scales.

This will also be a problem with rpms, which do support embedded
signatures natively. And the answer there will also clearly not be
“do not use embedded signatures”.

> >This will completely disallow digital signatures embedded in
> >the binary packages or per-executable signatures in their xattr
> >metadata for example, and that seems to me was completely ignored
> >last time around. I'd be very unhappy if at some point reproducible
> >builds were enforced and that'd mean we've painted ourselves into a
> >corner with other potential additions to the .deb that might not be
> >reproducible by design.
> 
> I think, not allowing unreproducible fields in .debs is a feature and a nice 
> design.
> 
> And yes, there has been years work on embedding signatures into debs, but I 
> dont see it as a problem that the result of this work is: don't do that, it 
> causes $these problems.  Detached signatures are pretty common everywhere.

Detached signatures have their place in various circumstances and
for various motives, they are not always the best option though. You
usually use detached signatures precisely because you cannot embed
them in the files you are signing. When you can, because the format
supports or allows it, it's almost always superior to embed them,
because then you don't easily lose track of those.

> Also, regarding embedded signatures, sure they are nice, but once we have 
> reproducible builds, they are also _way_ less meaningful, as the 
> reproducibility (and the signed source) make a even more useful statement: 
> with embedded signatures you still need to trust the signee that this binary 
> derived from the source he/she says it derives from. With reproducible builds 
> you can independently veriry that the binary indeed comes from this source.

I think trying to confront those two features would be very unwise.
They have different applications, and I'm sorry to say but while
reproducibility has some very nice attributes and implications, it
does not suddenly nullify everything else. It's a bit like saying that
signatures on source packages are way less meaningful because you
can download the sources and review them. You have to trust something
or someone.

Take the example I gave previously of a binary package detached from
an archive, just a .deb package laying around, either from an old
download or passed to you by someone. You have to *know* the origin of
the binary, otherwise you need to first start hunting down where this
binary was built, say Debian, one of its derivatives or even somewhere
else. And sure, once that's known, the user *might* possibly be able to
reproduce the build, but I don't see many (if not most) users being able
or willing to set up a reproducible build environment just to verify
where a binary was coming from (say my relatives). If you cannot or
wont do that, you need to trust the distribution, the remote server,
the network, the remote binary including any possible reproducible
information being correct. At that point you or a program might as well
have just verified an embedded signature.

Having the possibility to reproducibly build stuff is very nice, but
manye people will be trusting a tiny set of other people doing those
reproducible builds, because they don't fancy running a source based
distribution, building the world on each upgrade.

> >  * Some of the information in this file trigger my privacy alarm bells,
> >for example the Build-Path field.
> 
> While I don't really share your concerns here, as I think the situation now 
> is 
> worse: tools embedd this private data into binaries and not many people know 
> about this. So I think making the build path visible (and explaining why we 
> have to do this) is actually a step in the right direction, on the way to the 
> right fix: not embedding the build path at all.

They don't know because the embedded DWARF sections are compressed, so
a simple «strings» on the binaries does not reveal anything suspicious.
You need to use something like «objdump -W» or «readelf -w». Having a
lintian check that at least warns when finding paths like
«/home/[[:alnum:][:punct:]]+/» would go some way to make people aware
of the problem.

And embedding the path for every and each build is actually way, way
worse, because currently t

Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo

[Guillem Jover]

Hi!

On Sun, 2015-02-01 at 10:46:50 +0100, Jérémy Bobbio wrote:
> Guillem Jover:
> > Looking at .
> > 
> > Have you seen any actual problem to warrant the «Ensure stable order
> > of Checksums-* fields» commit? In principle the output order is
> > preserved from the input one.
> 
> I have seen the ordering differ, but I might have misunderstood the
> source of the problem. Unfortunately, the packages have since been
> tested again and I did not kept track of them. We can always stumble
> again on the problem later.

Ok.

> > And here's a quickish review of the dpkg-genbuildinfo changes, taking
> > into account that I'm looking at this as a general tool for recording
> > the build environment, not specific for just reproducible builds.
> 
> Guillem, thanks for your review! I had not asked for one yet, because we
> are still exercising this code and design decision on the archive. But I
> guess time has come. :)

No problem!

> >  * I'm still somewhat unconvinced that having byte-for-byte identical
> >container binary .deb packages is the ideal minimal reproducible
> >unit. This will completely disallow digital signatures embedded in
> >the binary packages or per-executable signatures in their xattr
> >metadata for example, and that seems to me was completely ignored
> >last time around. I'd be very unhappy if at some point reproducible
> >builds were enforced and that'd mean we've painted ourselves into a
> >corner with other potential additions to the .deb that might not be
> >reproducible by design.
> 
> This is a fair point. Having byte-for-byte identical .deb files does
> not prevent digital signatures. It only makes the process slighly more
> complex. One solution is to record the signatures in the buildinfo (or
> in another build product). It can then be copied as-is during rebuilds.

Hmm, the current thinking is to store signatures in a separate ar
member as a compressed tarball. This means encoding and embedding that
in the *.buildinfo file would probably bloat it quite a bit. Also
there's been for a long time this idea floating around of possibly
signing the binary packages on some steps of their processing, say
by the builder, then by the archive, so that one could always check
its provenance even when the binary packags is disconnected from an
archive. I guess the archive could at the same time mangle the
*.buildinfo file, but it all starts to look a bit fragile.

This also does not work for the per-executable signatures (or would
make it unbearably more complex), which although I don't see being
deployed in Debian, makes it conflict with each other.

One option I mentioned some time ago is to make something like
dpkg-deb generate a unique ID from its contents, instead of its entire
container, which could be recorded in the .buildinfo file, and that
would then be unaffected by this kind of non-reproducible by-design
artifacts.

> >  * Somewhat related to the above point and this new file. Timestamps
> >in some places (mainly on the actual file metadata, for example in
> >ar and tar containers) are actually very useful, because as long as
> >we don't have a stored recording of the build environment, it's the
> >next "best" approximation to that. Getting rid of those prematurely
> >makes us miss very valuable information for debugging and similar.
> >And yes, even replacing the current time with the changelog time is
> >missing information. This has also been dismissed previously (with,
> >I'd say, even a bit of contempt!).
> 
> Sorry if it felt contemptuous. You might have more experience than me
> about using these information for debugging.

I don't think this kind of information is used regularly for debugging,
because it's too broad and inaccurate, as it depends on the local clock
being correct, and on the upgrade habits of the sysadmin. But it's
easy to record and (assuming a working clock), it sets an upper time
bound, by which you know for sure if newer versions of other software
cannot have been used; and if you assume an up-to-date system, then you
can probably reproduce the environment to a pretty close state.

For example I'd be able to reproduce the environment for any of my
builds from the timestamp in the .deb ar members, but not from the
changelog, because sometimes I finish the release, but do not build
it until I'm about to upload, which might take days or weeks.

That's one of the reasons you have and will keep finding resinstance to
removing such timestamps from upstream projects, because even if in some
cases it might be out of old habits, this is in many others valuable
information that was there and you are pro

[Reproducible-builds] Preliminary review of dpkg-genbuildinfo

[Guillem Jover]

Hi!

Looking at .

Have you seen any actual problem to warrant the «Ensure stable order
of Checksums-* fields» commit? In principle the output order is
preserved from the input one.

And here's a quickish review of the dpkg-genbuildinfo changes, taking
into account that I'm looking at this as a general tool for recording
the build environment, not specific for just reproducible builds. Some
general comments first:

 * It would have been nice to get the .buildinfo stuff up for review
   in the debian-dpkg list too (besides the ftp-masters bug 763822).

 * I'm still somewhat unconvinced that having byte-for-byte identical
   container binary .deb packages is the ideal minimal reproducible
   unit. This will completely disallow digital signatures embedded in
   the binary packages or per-executable signatures in their xattr
   metadata for example, and that seems to me was completely ignored
   last time around. I'd be very unhappy if at some point reproducible
   builds were enforced and that'd mean we've painted ourselves into a
   corner with other potential additions to the .deb that might not be
   reproducible by design.

 * Somewhat related to the above point and this new file. Timestamps
   in some places (mainly on the actual file metadata, for example in
   ar and tar containers) are actually very useful, because as long as
   we don't have a stored recording of the build environment, it's the
   next "best" approximation to that. Getting rid of those prematurely
   makes us miss very valuable information for debugging and similar.
   And yes, even replacing the current time with the changelog time is
   missing information. This has also been dismissed previously (with,
   I'd say, even a bit of contempt!).

 * I'm not entirely sure if this really makes sense as a different
   file, but at least given that it's controlled by dpkg-buildpackage
   we can always fold it into dpkg-genchanges if we deem that's a
   better course of action. So it seems fine this way for now.

 * Some of the information in this file trigger my privacy alarm bells,
   for example the Build-Path field.


> diff --git a/debian/usertags b/debian/usertags
> index 0fc26f2..0419488 100644
> --- a/debian/usertags
> +++ b/debian/usertags
> @@ -65,6 +65,7 @@ dpkg-checkbuilddeps [DPKG-CHECKBUILDDEPS]
>  dpkg-deb [DPKG-DEB]
>  dpkg-distaddfile [DPKG-DISTADDFILE]
>  dpkg-divert  [DPKG-DIVERT]
> +dpkg-genbuildinfo[DPKG-GENBUILDINFO]

The pseudo-tag is only needed for old tools for legacy reasons. I
should just do a round of «bts retitle» at some point.

>  dpkg-genchanges  [DPKG-GENCHANGES]
>  dpkg-gencontrol  [DPKG-GENCONTROL]
>  dpkg-gensymbols  [DPKG-GENCSYMBOLS]
> diff --git a/man/dpkg-genbuildinfo.1 b/man/dpkg-genbuildinfo.1
> new file mode 100644
> index 000..626bf66
> --- /dev/null
> +++ b/man/dpkg-genbuildinfo.1
> @@ -0,0 +1,89 @@
> +.\" dpkg manual page - dpkg-genbuildinfo(1)
> +.\"
> +.\" Copyright © 2015 Jérémy Bobbio 

This probably needs to be completed with the © from the sources it's
borrowing from.


The changes to man/po/po4a.conf are missing.

> diff --git a/scripts/Dpkg/Control/FieldsCore.pm 
> b/scripts/Dpkg/Control/FieldsCore.pm
> index 8a5695b..75de185 100644
> --- a/scripts/Dpkg/Control/FieldsCore.pm
> +++ b/scripts/Dpkg/Control/FieldsCore.pm
> @@ -361,6 +373,11 @@ our %FIELD_ORDER = (
>  Vcs-Svn Testsuite), &field_list_src_dep(), qw(Package-List),
>  @checksum_fields, qw(Files)
>  ],
> +CTRL_FILE_BUILDINFO() => [
> +qw(Format Build-Architecture Source Binary Architecture Version
> +Build-Environment Build-Path),
> +@checksum_fields,
> +],

Probably pack all «Build-» fields together at the end after the checksum
ones, and leave Build-Environment as the last of those, as it's the one
taking the most vertical space.

>  CTRL_FILE_CHANGES() => [
>  qw(Format Date Source Binary Binary-Only Built-For-Profiles 
> Architecture
>  Version Distribution Urgency Maintainer Changed-By Description
> diff --git a/scripts/dpkg-buildpackage.pl b/scripts/dpkg-buildpackage.pl
> index a3cca87..018f37d 100755
> --- a/scripts/dpkg-buildpackage.pl
> +++ b/scripts/dpkg-buildpackage.pl
> @@ -158,6 +158,7 @@ my $since;
>  my $maint;
>  my $changedby;
>  my $desc;
> +my @buildinfo_opts;
>  my @changes_opts;
>  my @hook_names = qw(
>  init preclean source build binary changes postclean check sign done

The buildinfo hook name is missing in the @hook_names array.

> @@ -567,6 +570,25 @@ if ($include & BUILD_BINARY) {
>  withecho(@debian_rules, $buildtarget);
>  run_hook('binary', 1);
>  withecho(@rootcommand, @debian_rules, $binarytarget);
> +
> +if (-e "../$pv.dsc") {

Why is the buildinfo file tied to whether there's or not a source
package? I'd say it might make sense to not generate a buildinfo file
only if we are not building binary packages. To check for that use the
BUILD_ con

Re: [Reproducible-builds] Bug#762683: libdpkg-perl: add build flags that help with build reproducibility

[Guillem Jover]

Hi!

On Sat, 2014-09-27 at 10:47:09 +0800, Paul Wise wrote:
> On Fri, 2014-09-26 at 19:30 +0200, Guillem Jover wrote:
> > Meanwhile I've added support to dpkg-buildflags to honor feature area
> > settings from DEB_BUILD_OPTIONS, so that these can be enabled globally by
> > a builder for example for build tests and similar, but will be overridden
> > by DEB_BUILD_MAINT_OPTIONS though (in the same way build flags are).
> 
> Thanks.

Which requires a call to _parse_feature_area(), which I forgot to
mention in my previous mail. I've added it locally now.

> > I'm thinking that if this gets accepted by the project at large, then
> > it might even make sense to make it an error (-Werror=date-time) at
> > some later point, so the name would be wrong. How about 'timeless'?
> > Or can you think about some other more appropriate name?
> 
> I wasn't aware of the -Werror=date-time flag, thanks for pointing it
> out. We should indeed transition to making it an error at some point.

The -Werror=option is a generic variant to -Woption, it just turns any
such warning option into an error.

> > It would be nice if you could add documentation in the dpkg-buildflags
> > man page, a new .SS subsection, with something along the lines of the
> > first Hardening paragraph (I had already reworked the common parts out
> > of the Hardening subsection for other feature areas, so no need to
> > repeat those).
> 
> Updated each of the patches to include documentation updates too.

Cool, thanks! I've merged this now locally, it will be included in
dpkg 1.17.14. I ended up doing some very minor tweaks, like prefixing
the add_reproducible_flags with an _ to mark it as private, or
rewording a bit the commit message.

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Bug#762683: libdpkg-perl: add build flags that help with build reproducibility

[Guillem Jover]

Hi!

On Wed, 2014-09-24 at 20:24:15 +0800, Paul Wise wrote:
> Package: libdpkg-perl
> Severity: wishlist
> Tags: patch
> Usertags: sentpatch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: toolchain timestamps
> X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org

> The attached pair of patches add support for adding build flags that
> help with build reproducibility and add the -Wdate-time flag, which
> causes warnings when the time related macros are used. This has the
> potential to cause FTBFS in conjunction with -Werror so we are willing
> to do package rebuilds with this patch in order to test the theory.

Please, see

for the process to include it in the default set.

Although for now I'm fine merging this being disabled by default.
Meanwhile I've added support to dpkg-buildflags to honor feature area
settings from DEB_BUILD_OPTIONS, so that these can be enabled globally by
a builder for example for build tests and similar, but will be overridden
by DEB_BUILD_MAINT_OPTIONS though (in the same way build flags are).

> From d5538d91bb35d489ef2aaa5f9b8cf62fbed676cb Mon Sep 17 00:00:00 2001
> From: Paul Wise 
> Date: Wed, 24 Sep 2014 19:41:18 +0800
> Subject: [PATCH 1/2] Add a function for adding flags related to reproducible
>  builds.

> +sub add_reproducible_flags {
> +my ($self, $flags) = @_;
> +
> +# Features enabled by default for all builds.

Please change this to “# Default feature states.”, to make it
independent of their enabled/disabled state, I've already changed
the original string where this was copied from.

> +my %use_feature = (
> +);
> +
> +# Warnings when __TIME__ __DATE__ __TIMESTAMP__ macros used

And move this comment to the other patch.


> From 3644a69a3770eb1b7ed7c119c940ddd04c89b5e3 Mon Sep 17 00:00:00 2001
> From: Paul Wise 
> Date: Wed, 24 Sep 2014 19:41:46 +0800
> Subject: [PATCH 2/2] Warn about the use of __TIME__, __DATE__ and
>  __TIMESTAMP__ macros.

> @@ -229,9 +229,14 @@ sub add_reproducible_flags {
>  
>  # Features enabled by default for all builds.
>  my %use_feature = (
> + warntime => 1,
>  );

I'm thinking that if this gets accepted by the project at large, then
it might even make sense to make it an error (-Werror=date-time) at
some later point, so the name would be wrong. How about 'timeless'?
Or can you think about some other more appropriate name?

>  # Warnings when __TIME__ __DATE__ __TIMESTAMP__ macros used
> +if ($use_feature{warntime}) {
> + $flags->append('CPPFLAGS', '-Wdate-time');
> +}

I verified that this works with a standalone cpp, so it looks good.

It would be nice if you could add documentation in the dpkg-buildflags
man page, a new .SS subsection, with something along the lines of the
first Hardening paragraph (I had already reworked the common parts out
of the Hardening subsection for other feature areas, so no need to
repeat those).

Thanks,
Guillem

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds