Re: salsa.debian.org (git.debian.org replacement) going into beta

[Ximin Luo]

Holger Levsen:
> On Wed, Dec 27, 2017 at 05:03:11PM +0100, Holger Levsen wrote:
>> On Tue, Dec 26, 2017 at 08:56:18AM +, Chris Lamb wrote:
>>> I believe there are enough people in (or around) our community who dislike
>>> Github (for a variety reasons not productive to debate/repeat again here)
>>> that moving there would be problematic.
>>
>> ack.
>>
>>> However, as you imply, this would be the ideal time to somehow a bunch of
>>> non Debian-specific repos to something outside of the Debian namespace.
>>> It would be more convenient for me to to use salsa.debian.org, but I can
>>> really see the appeal of moving to, for example,
>>> https://gitlab.com/ReproducibleBuilds for anything not Debian-specific.
>>
>> gitlab.com is run+owned by Gitlab Inc., another company. Can you explain
>> why people seem to like this better than this other thing by another
>> company?
> 
> ping?
> 
> to me its just two companies. I'd like to understand why gitlab.com is
> better than github.com.
> 
> meanwhile, https://salsa.debian.org/salsa/support/issues/29 has been
> filed, which i have at least three issues with:
> 
> - i think the group name should be "reproducible-builds" not
>   "reproducible"
> - i'm not sure we should use salsa at all, see this very thread, we have
>   many non-Debian contributors…
> - why file this issue before this thread has ended?
> 

Sorry, I missed this thread. I was just responding to the fact that alioth is 
going away soon.

I picked "reproducible" because that's what the group is called on alioth.

I don't have an opinion on "using something else than salsa.debian.org" but if 
we don't decide before alioth gets decommissioned we still have to move the 
repos *somewhere*.

The ticket is just for squatting the name, it doesn't prevent any other 
efforts. I was not planning to migrate the git repos (and delete the old ones) 
without asking people further, of course.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 139's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/139/

Feel free to commit fixes directly to drafts/139.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 137's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/137/

Feel free to commit fixes directly to drafts/137.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/ocaml_4.05.0-10.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#881937: diffoscope: Doesn't show differences in .deb's control.tar.xz (regression?)

[Ximin Luo]

Axel Beckert:
> [..]
> 
> Even "--fuzzy-threshold 0" to disable any "fuzzy-matching" (whatever
> that is in the context of diffoscope) didn't make a difference.
> 

Actually, the problem is the other way round - control.tar.{gz,xz} have 
different names and the contents aren't similar enough (I guess because they 
are compressed differently).

If you give --fuzzy-threshold 400 (i.e. >212, as mentioned in your log) then it 
works.

In the meantime, I'll work on a fix that "just works" and doesn't require the 
user to pass that flag manually.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#879011: diffoscope: zipinfo diff shows warning differences that are due to temporary file names

[Ximin Luo]

Mike Hommey:
> On Thu, Oct 19, 2017 at 07:15:35PM +0900, Mike Hommey wrote:
>> Attached is my work in progress. I won't finish it right now because
>> it's getting late and I don't have all the optional tools to run the
>> test suite.
>>
>> It's worth noting that this changes the output, and the notable
>> difference is that the information whether files are stored or deflated
>> is lost. It doesn't seem at first glance that libarchive exposes that,
>> so that might be a showstopper :-/
> 
> Attached here is a full patchset that passes all zip-related tests. Please
> take a look at the changes in output, whether this is wanted or not.
> 

I've fixed it a different way:

https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?id=25fee28c8b29dbc66cd7bb57a2ab427651050c23

I didn't notice any slowdown that might have occurred due to the lack of random 
access to the file.

Let me know if this doesn't fix the original issue. I agree the 
stderr-to-stdout-when-tty is ridiculous. Funnily enough the behaviour isn't 
present when `-v` is enabled...

(I have applied 0001.patch since that's a separate issue, so thanks for that 
too.)

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 135's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/135/

Feel free to commit fixes directly to drafts/135.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/r-base_3.4.2.20171120-1.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#882511: dpkg-buildpackage: should allow caller to force inclusion of source in buildinfo

[Ximin Luo]

Package: dpkg-dev
Version: 1.19.0.4
Severity: wishlist
Tags: patch

Dear Maintainer,

dpkg-buildpackage currently does not automatically list the source .dsc nor
its hash in the call to dpkg-genbuildinfo when doing a binary-only build. This
is understandable because in a binary-only build, dpkg-buildpackage does not
have any concept of a source package and therefore does not know (and cannot
verify) if the working tree was actually generated from any .dsc or not.

However, the caller knows this information, and it is useful for reproducible
builds to track exactly which (i.e. hash-wise) source code generates which
binary packages. So it should be possible for the caller to tell
dpkg-buildpackage, "yes please do include the .dsc hash in the buildinfo, I am
telling you it is correct, you can assume this safely".

Tools like sbuild/pbuilder could then do this, as well as users or rebuilders.

The attached patch implements this in the simplest way possible. It allows the
caller to run something like:

  $ dpkg-buildpackage --no-sign -b --buildinfo-option=--build=full

The resulting $pkg_$ver_$arch.buildinfo then contains the .dsc and its hash.

However this requires the caller to know which option to pass, which would 
either be

  --buildinfo-option=--build=full
  --buildinfo-option=--build=any,source
  --buildinfo-option=--build=all,source

depending on whether the original build request (to dpkg-buildpackage) was a 
-b, -B, or -A.

For this reason, it may be better (more usable) to add a 
--force-source-in-buildinfo
flag (or similar name) and when this is switched on, do this instead:

-push @buildinfo_opts, "--build=$build_types" if build_has_none(BUILD_DEFAULT);
+push @buildinfo_opts, "--build=$build_types,source" if 
build_has_none(BUILD_DEFAULT);

Let me know if you like this idea and I'll be happy to implement that instead of
the attached patch.

X

-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 
'testing-debug'), (300, 'unstable'), (200, 'experimental'), (1, 
'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8), LANGUAGE=en_GB:en 
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dpkg-dev depends on:
ii  binutils  2.29.1-8
ii  bzip2 1.0.6-8.1
ii  libdpkg-perl  1.19.0.4
ii  make  4.1-9.1
ii  patch 2.7.5-1+b2
ii  perl  5.26.1-2
ii  tar   1.29b-2
ii  xz-utils  5.2.2-1.3

Versions of packages dpkg-dev recommends:
ii  build-essential  12.4
ii  clang-4.0 [c-compiler]   1:4.0.1-8
ii  fakeroot 1.22-2
ii  gcc [c-compiler] 4:7.2.0-1d1
ii  gcc-7 [c-compiler]   7.2.0-16
ii  gnupg2.2.2-1
ii  gnupg2   2.2.2-1
ii  gpgv 2.2.2-1
ii  libalgorithm-merge-perl  0.08-3

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2017.08.28

-- no debconf information

-- debsums errors found:
debsums: changed file /usr/bin/dpkg-buildpackage (from dpkg-dev package)
diff --git a/scripts/dpkg-buildpackage.pl b/scripts/dpkg-buildpackage.pl
index f759ba4a6..2250403db 100755
--- a/scripts/dpkg-buildpackage.pl
+++ b/scripts/dpkg-buildpackage.pl
@@ -27,6 +27,7 @@ use Cwd;
 use File::Temp qw(tempdir);
 use File::Basename;
 use File::Copy;
+use List::Util qw(none);
 use POSIX qw(:sys_wait_h);
 
 use Dpkg ();
@@ -574,7 +575,9 @@ if (build_has_any(BUILD_BINARY)) {
 
 run_hook('buildinfo', 1);
 
-push @buildinfo_opts, "--build=$build_types" if build_has_none(BUILD_DEFAULT);
+if (none { index($_, '--build=') == 0 } @buildinfo_opts) {
+push @buildinfo_opts, "--build=$build_types" if 
build_has_none(BUILD_DEFAULT);
+}
 push @buildinfo_opts, "--admindir=$admindir" if $admindir;
 
 run_cmd('dpkg-genbuildinfo', @buildinfo_opts);
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Please review the draft for week 133's blog post

[Ximin Luo]

Holger Levsen:
> On Thu, Nov 16, 2017 at 10:06:00AM +0000, Ximin Luo wrote:
>> Ping, are you ready? I don't see any commits from you in the log...
> 
> not yet & on a very bad internet connection atm.
> 
> i will publish 133 once it's ready.
> 

Hey, just a reminder that 133 is still not published.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> By the way: is there a macro or combination of macros which *default* value 
> can be replaced in the use of __FILE__ without caussing regressions?
> 
> Because if that's the case it's easier to convince upstream people that 
> changing the usage goes in favor of reproducibility without using strange up-
> to-the-builder definitions.
> 

Unfortunately, not that I know of.

However, let me try to explain why BUILD_PATH_PREFIX_MAP is not as strange as 
you might think.

GCC already supports a flag -fdebug-prefix-map whose purpose is to do the same 
thing for debugging symbols. You can give multiple maps, to map different parts 
of your source tree to different other source directories. So maybe one could 
give a map like this:

1. "PWD/" -> "/usr/src/debug/$mypackage"
2. "PWD/tests" -> "./tests"

if one wanted to generate debugging information for your main source code, to 
point to installed-source-code at /usr/src/debug/$mypackage; but since one 
doesn't install the tests anywhere, it would be convenient for developers to 
have the debuginfo of tests point to ./tests instead, so they can just run "gdb 
-d." if stuff fails.

What I'm suggesting for QT tests would be exactly analogous to this 
already-supported use case for debuginfo. dpkg gives a default map 
corresponding to (1), and since QT tests use __FILE__ for other purposes, you 
give a second map corresponding to (2) above.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> [..]
> 
> Yes, we do understand that your workaround solves the issue, but we do also 
> understand that we should not be using this workaround in the first place.
> 
> Guillem: the thread is long, but be sure that we Qt/KDE maintainers consider 
> that this change will be insta-RC I'm afraid.
> 

Guillem: the TL;DR is that some QT tests fail (with our reproducibility GCC and 
dpkg patch) because they depend on __FILE__ to point to a real path, but the 
patched dpkg rewrites it in all packages so that a prefix $PWD changes to 
$srcpkg-$version, which doesn't exist.

A simple fix on the dpkg side would be to instead rewrite $PWD to ".", much 
like -fdebug-prefix-map is done currently. However this loses some information 
and makes it hard to distinguish different packages that might share filetree 
structure. But that is the best fallback option IMO, if we decide that we don't 
want to auto-break QT tests for using  __FILE__ (IMO, inappropriately).

> Xi: you have found a *wonderful* way to find where bugs are, please try to 
> fix 
> the relevant code and not paper over it, because in the Qt case it is not a 
> bug on our side.
> 

I pointed to the various C standards documents, as well as documentation from 
multiple compilers, stating that __FILE__ is the "name of the source file" and 
in no way guarantees that the expansion can later be re-used as the path to an 
actual file. GCC documentation even explicitly states the expansion is 
arbitrarily chosen by the implementation of the preprocessor, and is explicitly 
"not [..] the input file name argument".

So I do consider this a bug in the QT test suite.

The ideal solution would be to not use __FILE__ - that has numerous other 
benefits as well. But if this is too complex to change, I also suggested a 
1-line addition to d/rules - which I agree would be "papering over" the issue. 
However, it's a simple 1-line change so I don't understand why there is so much 
resistance to it.

There are several other possible solutions, all of which are low-cost and 
unintrusive, and could be done in a QT build helper in one single place:

- define a custom macro, QT_TEST_SOURCE_BASE, and set that in the test build 
scripts, instead of using __FILE__
- export BUILD_PATH_PREFIX_MAP="$BUILD_PATH_PREFIX_MAP:tests=$BASEDIR/tests"
- symlink "$srcpkg-$version" -> "."

I would be very happy to send you the patches myself if you don't want to do 
the work, since writing 1-line patches to a few QT projects, costs far less 
time than patching 1800 packages across Debian.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Please review the draft for week 133's blog post

[Ximin Luo]

Holger Levsen:
> On Tue, Nov 14, 2017 at 01:34:00PM +0000, Ximin Luo wrote:
>> This week's blog post draft is now available for review:
>> https://reproducible.alioth.debian.org/blog/drafts/133/
> [...] 
>> I'll wait at least 24 hours from the time of this email for any comments, 
>> and if everything is good then I will publish it soon after that.
> 
> please hold off publishing this blog for another 24h.
> 

Ping, are you ready? I don't see any commits from you in the log...

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> On miércoles, 15 de noviembre de 2017 20:35:00 -03 Ximin Luo wrote:
>> Lisandro Damián Nicanor Pérez Meyer:
>>> I was wanting to write this on a keyboard, but let me try on the phone and
>>> see if we can all agree in one point.
>>>
>>> There is one *excellent* method to settle this once and forever: submit
>>> the
>>> __FILE__ patch to GCC devs. If it gets accepted then Qt and whatever
>>> project is affected should change it.
>>>
>>> If it does not gets accepted then it's a Debian regression.
>>>
>>> Until that is done I will consider this a non-Qt bug but a GCC regression.
>>> Show me that GCC upstream devs agree with you Xi and I'll happily forward
>>> the bug to Qt upstream.
>>
>> That sounds perfectly fair to me, and in line with my existing plans. Thanks
>> for your patience, and I will ping back hopefully with good news in a few
>> months.
> 
> By the way: I will consider this as a valid patch if it changes the __FILE__ 
> **default** behavior, not something that can make it change at the will of 
> the 
> builder (like an environmet variable or such).
> 
> Having the chance of changing the meaning does not means it can be a 
> regression.
> 

The GCC patch (neither the previous nor the planned version) does not change 
the default behaviour of __FILE__, and was never intended to. Instead, it gives 
users the ability to rewrite __FILE__, more specifically a prefix of it.

There is a separate patch to dpkg that enables this ability for all packages, 
in the same way that SOURCE_DATE_EPOCH is enabled. Guillem the dpkg maintainer 
has previously indicated that he's happy to take the patch, once GCC accepts 
their end of it.

It's a combination of these two patches that would cause these QT tests to 
fail. The reason it fails, is because we specifically map $PWD to a 
non-existent path. I suggested various ways around this. One of the 
suggestions, was to add an extra mapping to map $PWD/tests back to $PWD/tests 
(or just ./tests), overriding the earlier non-existent mapping. I think this is 
the cleanest suggestion - assuming that tests reside in the same directory, and 
away from the main source code. Kai Pastor over on bug #876934 indicated that 
this would probably work for openorienteering-mapper.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> I was wanting to write this on a keyboard, but let me try on the phone and
> see if we can all agree in one point.
> 
> There is one *excellent* method to settle this once and forever: submit the
> __FILE__ patch to GCC devs. If it gets accepted then Qt and whatever
> project is affected should change it.
> 
> If it does not gets accepted then it's a Debian regression.
> 
> Until that is done I will consider this a non-Qt bug but a GCC regression.
> Show me that GCC upstream devs agree with you Xi and I'll happily forward
> the bug to Qt upstream.
> 

That sounds perfectly fair to me, and in line with my existing plans. Thanks 
for your patience, and I will ping back hopefully with good news in a few 
months.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> [..]
> 
>> In summary: in no document or standard, does it guarantee or imply
>> that __FILE__ can be taken to represent a real filesystem path.
>> Applications relying on this behaviour are broken and should not be
>> upset when things don't work. As documented in multiple places,
>> __FILE__ only has a very loose meaning, my patch fits within this
>> loose meaning, and it is intended mostly for error messages.
> 
> ... this is your own interpretation.  Even if __FILE__ has a loose
> meaning, you are still changing that small bit of meaning it has,
> without even thinking twice.  Also, where it is written that __FILE__
> is "only for error messages"?
> 

The loose meaning, indicates that programs should not rely on it for other 
meanings, and should be open to fixing themselves if implementations change 
their specifics. This is a general principle that applies to all text in all 
standards, it is not "my own interpretation".

And stop twisting my words. You literally just misquoted me to advance your own 
point. I said "intended mostly for error messages":

https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
"__FILE__ and __LINE__ are useful in generating an error message [..]"

https://msdn.microsoft.com/en-us/library/027c4t2s.aspx
"Without /FC, the diagnostic text would look similar to this diagnostic text:"

The only examples I can find in compiler documentation of __FILE__ usage, is 
for error messages. There are no examples of any other usage. That strongly 
suggests it was originally intended for error messages and never intended to 
support file lookups. That's not the same thing as saying "it's intended only 
for error messages", but it does strongly imply that you shouldn't be surprised 
if your file lookups break with minor changes like my patch, and it certainly 
gives you no right to go off on a rant about me "breaking" stuff.

http://c0x.coding-guidelines.com/6.10.8.html The presumed name of the current 
source file [..]
https://software.intel.com/en-us/node/524489 Defined as a character string 
literal containing the name of the source file.
https://www.ibm.com/support/knowledgecenter/SSAE4W_9.1.1/com.ibm.etools.iseries.langref.doc/ilcrefer13.htm
 A string literal representing the name of the file being compiled.

You can even find threads online confirming __FILE__ is implementation-specific 
and not required by any standard to be a full path. You can even find threads 
of people complaining about the fact that __FILE__ sometimes has a full path, 
sometimes not - e.g. depending on what options you pass to the Microsoft 
compiler. Go do your own research about it.

You need to get your facts straight before going on an ill-tempered rampage 
accusing me in bad faith of "not having a discussion" and being in "my own echo 
chamber". And stop pretending that my arguments are "my personal beliefs", they 
are not. And stop accusing me of blindly changing stuff against the standards, 
without knowing what the standards actually are. And stop accusing me of making 
my mind up before the discussion, when it is you that seems to have made up 
your mind before the discussion that I was "blindly" making changes and not 
thinking through consequences, I did think those through perfectly well thank 
you very much.

And stop diverting the thread to making meta-comments about my word usage. It 
makes it seem like you're trying to make me look bad in front of an audience. I 
don't like playing those sorts of games, which is why I'm ignoring various 
other things you've been saying.

And make some concrete proposals.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> [..]
> 
> Xi: you also mentioned that having to file hundreds of patchs seems 
> impossible. Well, it seems so, but it is actually not that necessary. Please 
> allow me to explain the idea.
> 

Thanks for being less inflammatory than Pino.

I agree that eventually all projects should move away from using __FILE__. This 
BUILD_PATH_PREFIX_MAP variable is only a stepping stone to that, just like 
SOURCE_DATE_EPOCH was a stepping stone to less projects using __DATE__ and 
__TIME__. It allows people to see the obvious benefit of a reproducible build, 
by actually achieving a large amount of reproductions, today. We did not need 
to file mass bug reports for __DATE__ and __TIME__ with SOURCE_DATE_EPOCH.

> What you can do here is starting by documenting/blogging about bad use cases 
> so people have something to read when bugs arrive.
> 
> [..]

You're implying that QT's use of __FILE__ for tests, as being discussed in this 
thread, is a "good use case" - and that it is *other people* with "bad use 
cases" that we should be spending a lot of time and effort to reach out to.

That's not how I see it. As I pointed out various times already, the use of 
__FILE__ here is *also not appropriate*. You can consider these emails from me 
now, as "documenting/blogging" about "bad use cases".

In summary: in no document or standard, does it guarantee or imply that 
__FILE__ can be taken to represent a real filesystem path. Applications relying 
on this behaviour are broken and should not be upset when things don't work. As 
documented in multiple places, __FILE__ only has a very loose meaning, my patch 
fits within this loose meaning, and it is intended mostly for error messages.

The BUILD_PATH_PREFIX_MAP variable works around a lot of "bad use cases", but 
it doesn't cover this particular case. Some minor tweaks are needed to cover 
it, and I suggested them already. But you guys continue to push the position 
"we're doing nothing wrong, it's other people doing stuff wrong, go talk to 
them instead".

I'm sorry but it's not a convincing position for me to agree with.

At the end of the day, all of these cases, including yours, ought to be fixed 
to not use __FILE__ at all. BUILD_PATH_PREFIX_MAP doesn't prevent the fix, it 
just means we can achieve many many real reproductions today without having to 
wait. That's a good thing.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> In data martedì 14 novembre 2017 11:14:00 CET, Ximin Luo ha scritto:
>> You're using __FILE__ inappropriately, none of the documentation
>> guarantees or implies that you can access __FILE__ as a real
>> filesystem path. "Surely for relative paths" is your justification
>> for your own broken behaviour.
> 
> Again, this is your own echo chamber: "__FILE__ is broken, everybody
> using it is broken no matter what".
> 

It's not my "own echo chamber", I pointed you to lots of docs that confirm your 
usage of __FILE__ is not appropriate here.

Adrian Bunk also mentioned the C89 to me on IRC yesterday. Again, the wording 
gives no guarantee that __FILE__ should represent a real filesystem path.

If my previous words were a bit terse I am sorry for that, but I don't 
appreciate comments like "Any of your solutions get a big, fat, and giant nope" 
after trying to explain the problem and present you with no less than 4 
alternative simple unintrusive solutions.

>> You can either accept my one line patch suggestion, or fix it some
>> other way.
> 
> I am not interested in working around broken changes introduced blindly
> for very doubtful reasons.
> 
>> I'm not going to change the GCC patch, it does nothing wrong.
> 
> Let me add also another POV to this approach: do you really expect
> Debian to carry this important diversion for GCC upstream?  I really
> doubt GCC will accept this.
> 

I'm in the process of getting the patch into GCC. We certainly don't intend to 
keep this as a Debian-specific thing. [1] If they don't accept it, you don't 
need to patch your package - but I'd say the use of __FILE__ is still not the 
best, since no documentation implies it can be used to find files, and all the 
examples only mention error messages.

And to be clear, in case Holger's earlier messages were missed, the FTBFS is 
currently only on tests.r-b.org and not on the official Debian archive.

X

[1] Although interestingly, NetBSD have a GCC patch that was written by us (dkg 
specifically) for a related purpose, that was not accepted into GCC. But they 
use it for reproducibility purposes.

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 133's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/133/

Feel free to commit fixes directly to drafts/133.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Lisandro Damián Nicanor Pérez Meyer:
> [..]
> 
> Sorry, it is not realistic to force us to have a delta from upstream for no 
> direct gain.
> 

None of the solutions I suggested involve patching upstream, they would be done 
in d/rules.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> [..]
> 
>> Well let's take a look at the standards:
>> [...]
> 
> Even with different wordings, both describe basically the same
> behaviour.  And yes, none of them says that __FILE__ is a full path,
> but surely for relative paths the combination of $srcdir + __FILE__
> will give me the path to the source.
> 

Whatever, I don't care about having such a pointless long discussion. You're 
using __FILE__ inappropriately, none of the documentation guarantees or implies 
that you can access __FILE__ as a real filesystem path. "Surely for relative 
paths" is your justification for your own broken behaviour.

And that's just for GCC, IBM and Microsoft both use the term "name of the file" 
which don't even suggest any sort of path.

You can either accept my one line patch suggestion, or fix it some other way. 
I'm not going to change the GCC patch, it does nothing wrong.

Not going to spend any more time on this.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> [..]
> 
> This is not an annoyance, it is the crux of the problem!  __FILE__ is
> something standard, with a very well defined behaviour, upon which
> people rely on: you cannot trash it from one day to another like this,
> saying "too bad" to all its users, even those using it appropriately.
> It does not simply work.
> 
>> but the point is that the previous use does not generalise well.
> 
> To be honest: I already said multiple times that abuses of __FILE__
> *must* be fixed.  However, there are valid use cases, and in this case
> the only generalization is done by you, who declared __FILE__ as
> "absolutely BAD".
> 

It's an objective fact that tests using __FILE__ are unable to be generalised 
to be run outside of the build.

As Lisandro said in the other comment, "We do *not* want to run the tests 
outside the build."

As I said in my previous comment, "For example if you try a program that works 
fine when compiled natively but doesn't work when cross-compiled, that is a 
failure to generalise and one can either say "not supported, stop breaking us" 
or one can try to generalise one's program."

It's also an objective fact that we're encouraging post-install tests such as 
autopkgtest. Pre-install tests (dh_auto_test) are good because of a tighter 
feedback loop, post-install tests are good because they are more accurate, they 
test directly what users install. For example, that installation paths were 
actually correct.

>> It's not a sledgehammer solution, it's tweakable and we can try to
>> use its flexibility to un-break your tests.
> 
> No, it *is* a sledgehammer solution, since you are not even checking
> what __FILE__ is used for, and thus blindly changing it without any
> pre-research on its effect.  Again, this is not the correct approach
> to fix the issue at hand.
> 

There are quite a lot of packages that use __FILE__ so forgive me for not 
checking every single use-case of it.

My patch fixes 1800 packages, and the amount of extra FTBFS it caused (such as 
this) is a blip in the stats, and a very minor one at that - failing tests, 
because a file was not found. Give me a break.

(Yes it is RC, but technically it is minor and very easy to fix, but you're 
denying us the fix because you seem to feel that the standards are on your 
side. Well let's take a look at the standards:

- https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
  "This is the path by which the preprocessor opened the file, not the short 
name specified in ‘#include’ or as the input file name argument. "
  Nope, no guarantee of absolute path here. The preprocesser could have cd'd to 
a random directory and then opened the file by a relative path.

- https://msdn.microsoft.com/en-us/library/027c4t2s.aspx
  "__FILE__ The name of the current source file. __FILE__ expands to a 
character string literal. To ensure that the full path to the file is 
displayed, use /FC (Full Path of Source Code File in Diagnostics). This macro 
is always defined."
  No guarantee of absolute path here either. It even says to guarantee full 
path you need to pass a specific compiler switch.

So, get off your high horse about "standards" and "well-defined". I really 
didn't want to pull this card because I think it is missing the point, but you 
seem to hold onto it with particular religious fervour.)

>> So, what do you think of this alternate solution, that I suggested:
> 
> Any of your solutions get a big, fat, and giant nope.
> 

You're forcing us to choose between fixing 1800 packages minus your QT 
packages, or fixing nothing. I ask you to be slightly less absolutist. The 
solution I proposed is 1-line and should cause basically no disruption to 
anything else.

>> If this isn't suitable to you, how else do you suppose we should try
>> to make builds independent of (i.e. not embed) the build path?
>> Suggestions are welcome.
> 
> I already wrote it in my email, and I repeat it again:
> 
> | No, the solution is:
> | a) *not* break what __FILE__ means
> | b) remove the misuses of __FILE__ in packages (not the case of
> |QFINDTESTDATA)
> 

This will require us to patch hundreds of packages, and isn't realistic. 
Please, you try sending hundreds of patches, then I will take your "solution" 
more seriously.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> [..]
> 
> A better approach here is to work on removing the invalid & abusing
> usages of __FILE__ from packages, just like it was done for __DATE__.
> 

We in fact did not do the latter because it was easier to implement 
SOURCE_DATE_EPOCH to fix the expansion of __DATE__, rather than patch 400 
packages.

With __FILE__ I believe the number is similar, but I didn't yet count exactly 
how many. (Our BUILD_PATH_PREFIX_MAP patch fixes 1800 packages, but only some 
of these are due to __FILE__.)

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876901: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Pino Toscano:
> [..]
> 
> No, the solution is:
> a) *not* break what __FILE__ means
> b) remove the misuses of __FILE__ in packages (not the case of
>QFINDTESTDATA)
> 
>> I am not "blaming the user", but pointing to the fact that __FILE__
>> is being used in a surprising way here, which is not good for
>> reproducible builds.
> 
> What I see it is happening here is: you (= people working on
> reproducible builds) see __FILE__, and the problems that arise from its
> abuse; to overcome this issue, you use the sledgehammer solution,
> basically changing what __FILE__ means, and thus breaking even valid
> use cases.  Sorry, but I do not see how this is useful.
> 
> A better approach here is to work on removing the invalid & abusing
> usages of __FILE__ from packages, just like it was done for __DATE__.
> 
>> An analogy would be to write your program to execute something at
>> time "__DATE__ + 30 seconds". This is obviously ridiculous, but works
>> "by accident" if done inside a test case.
> 
> This is clearly a misuse, and thus it must be fixed.  OTOH, the
> comparison with __FILE__ is not appropriate.
> 

Why's it not appropriate? If you ever want to write tests to be runnable 
outside the build, e.g. with autopkgtest, then you're going to have to not use 
__FILE__ anyway. (Assuming you install the tests somewhere, rather than running 
the whole build again.)

I can understand that breaking something that used to work is annoying, but the 
point is that the previous use does not generalise well. I don't really want to 
get into holy wars about the definition of "validity", I just ask you to 
consider generality. For example if you try a program that works fine when 
compiled natively but doesn't work when cross-compiled, that is a failure to 
generalise and one can either say "not supported, stop breaking us" or one can 
try to generalise one's program.

It's not a sledgehammer solution, it's tweakable and we can try to use its 
flexibility to un-break your tests. So, what do you think of this alternate 
solution, that I suggested:

Myself:
> If all the test files reside underneath the same directory, like tests/, you 
> could:
> 
> 4. export BUILD_PATH_PREFIX_MAP="$BUILD_PATH_PREFIX_MAP:tests=$BASEDIR/tests".
> 
> This should make the tests pass, whilst keeping our "$srcpkg-$version" 
> mapping for the non-test files.

Explanation: the map is parsed right-to-left so the later map will be activated 
first for files underneath tests/ and this mapping will be applied, rather than 
the default mapping set by our patched dpkg.

This can be done generically in a build helper or something, rather than 
per-package.

If this isn't suitable to you, how else do you suppose we should try to make 
builds independent of (i.e. not embed) the build path? Suggestions are welcome.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Ximin Luo:
> [..]
> 
> (maybe there are other options)
> 

If all the test files reside underneath the same directory, like tests/, you 
could:

4. export BUILD_PATH_PREFIX_MAP="$BUILD_PATH_PREFIX_MAP=tests=$BASEDIR/tests".

This should make the tests pass, whilst keeping our "$srcpkg-$version" mapping 
for the non-test files.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Ximin Luo:
> Ximin Luo:
>> [..]
>>
>> (maybe there are other options)
>>
> 
> If all the test files reside underneath the same directory, like tests/, you 
> could:
> 
> 4. export BUILD_PATH_PREFIX_MAP="$BUILD_PATH_PREFIX_MAP=tests=$BASEDIR/tests".
> 

Argh, I mean of course:

4. export BUILD_PATH_PREFIX_MAP="$BUILD_PATH_PREFIX_MAP:tests=$BASEDIR/tests".
   ^
colon, not equals

> This should make the tests pass, whilst keeping our "$srcpkg-$version" 
> mapping for the non-test files.
> 
> X
> 


-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: QFINDTESTDATA uses __FILE__

[Ximin Luo]

Adrian Bunk:
> Control: reassign -1 qtbase5-dev
> Control: reassign 876917 qtbase5-dev
> Control: reassign 876933 qtbase5-dev
> Control: forcemerge -1 876917 876933
> Control: retitle -1 QFINDTESTDATA uses __FILE__
> Control: severity -1 normal
> Control: affects -1 src:karchive src:ki18n src:kcodecs src:kparts
> thanks
> 
> The problem is the following implementation in
> /usr/include/x86_64-linux-gnu/qt5/QtTest/qtestcase.h:
> # define QFINDTESTDATA(basepath)\
> QTest::qFindTestData(basepath, __FILE__, __LINE__)
> 
> With the patched gcc in the unstable reproducible builds setting
> __FILE__ to something other value, this does no longer find the
> test data.
> 
> I do not really see any reason for blaming the users here for using
> a documented public Qt API for accesssing test data in the source
> directory:
>   http://doc.qt.io/qt-5/qtest.html#QFINDTESTDATA
> 
> I've added reproducible-builds@lists.alioth.debian.org to Cc for giving 
> input what a reproducible and portable implementation might be for Qt.
> 

Our patched dpkg tells GCC to map $PWD to "$srcpkg-$version" when expanding 
__FILE__. That is the source of the problem, because this path doesn't exist at 
test-time. You have the following options:

1. Unset BUILD_PATH_PREFIX_MAP. This is not great because things that use 
__FILE__ will become unreproducible.

2. Symlink "$srcpkg-$version" -> ".", so that the files can be accessed under 
the paths that __FILE__ was expanded to.

3. Set BUILD_PATH_PREFIX_MAP to map $PWD to . instead. You do this by doing 
`export BUILD_PATH_PREFIX_MAP=$BUILD_PATH_PREFIX_MAP:.=$PWD`, then the tests 
should work. This makes any other uses of __FILE__ in this package, 
inconsistent with other Debian packages (built with our patched dpkg).

(maybe there are other options)

I chose "$srcpkg-$version" because it provides some extra information, and 
allows one to distinguish files in different packages. Currently, 
dpkg-buildflags(1) sets -fdebug-prefix-map= to ".", so propsal (3) would 
actually be consistent with that. However I do think "$srcpkg-$version" is 
better because it's more informative. More generally, we don't want 
__FILE__-using tests to dictate how we should remap build paths *in general* in 
Debian.

The problem stems from the fact that we assume __FILE__ represents a build-time 
path and not a run-time path, so we rewrite it indiscriminately with 
BUILD_PATH_PREFIX_MAP.

This assumption is broken in the specific case where you have some source code 
that uses __FILE__, that are specifically only meant to be run at build-time, 
so that they are in fact run-time paths (that are also build-time paths). The 
assumption is fine in all other cases.

Therefore, the only solution to fix this problem, that also preserves 
reproducible builds, is to make those tests *not use __FILE__*. Or option (2), 
which makes the non-existent rewritten paths, into an actually-existing 
build-time path.

I am not "blaming the user", but pointing to the fact that __FILE__ is being 
used in a surprising way here, which is not good for reproducible builds. An 
analogy would be to write your program to execute something at time "__DATE__ + 
30 seconds". This is obviously ridiculous, but works "by accident" if done 
inside a test case.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 131's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/131/

Feel free to commit fixes directly to drafts/131.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/r-base_3.4.2-2.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#877813: reprotest: regression between 0.7.1 and 0.7.2

[Ximin Luo]

gregor herrmann:
> [..]
> 
> So I guess I'll stay with excluding domain_host for now.
> 
> [..]
> 
> It looks like currently both 0 and 1 cause problems :)
>  
Yes, I've experienced this too. :( I'm hitting what I think might be a bug with 
nsenter(1), but I have a potential workaround. It'll take a while to code up 
though.

Unfortunately the code from current git, causes some leftover bits on your 
system. You probably want to, in this order:

1. check no reprotest processes are running, exit / ctrl-c them if so
2. check the output of `ps -elf | grep autopkgtest-virt` to make sure no 
autopkgtest-virt-* processes are running. this is a side-effect of 
--no-clean-on-error I didn't spot. if there are, just `kill -INT` them etc 
until they go away.
3. check the output of `schroot -la` to make sure there's no extra chroot 
sessions, if there are you can delete them after performing the previous step
4. check the output of `findmnt` to make sure you don't have extraneous bind 
mounts on /etc/host, if there are you can just `sudo umount` them.
5. then you can delete any /tmp/autopkgtest.** and /tmp/reprotest.** left over

Sorry about the bother. I'll do some more testing before releasing this, of 
course.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#877813: reprotest: regression between 0.7.1 and 0.7.2

[Ximin Luo]

gregor herrmann:
> [..]
> 
> I've now built a package from git HEAD (3efb86f), and .. hm .. I guess
> this bug is fixed, it's just that I'm running into another issue
> which seems related to another recent commit (the domain_host
> feature):
> 
> [..]
> 
> The recommended `echo 1 > /proc/sys/kernel/unprivileged_userns_clone'
> (as root on the host) doesn't help.
> 

I understand it's not nice to keep having to add -disable flags on every new 
reprotest release, sorry about that.

Does it work if you give --vary=domain_host.use_sudo=1? (I think it should, if 
you have root inside the chroot.)

Are you running reprotest on a developer machine or a CI platform or something? 
If I understood more about how people expect to use reprotest, maybe I could 
make these newer features smoother to use. My motivation for defaulting 
use_sudo to 0, was to allow non-root users to "try it out". But perhaps I 
should just default it to 1, if it causes problems for most people.

> Adding -domain_host to --variations worksaround the issue as
> expected, and confirms that the original issue is indeed fixed.
> 
> Yay! Thanks again.
> 

Welcome!

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#877813: reprotest: regression between 0.7.1 and 0.7.2

[Ximin Luo]

gregor herrmann:
> [..]
> 
> So currently I have:
> 
> - reprotest called as 
>   env -u TMPDIR -- reprotest --variations=+all,-build_path,-user_group 
> --verbosity 1 . -- schroot default 2>&1 | [..]
> 

Thanks, this helped me reproduce the bug locally. I was having trouble before, 
because the bug only occurs when fileordering is switched on but user_group is 
not switched on.

The bug is because disorderfs was being run as root inside the schroot, but the 
build was being run as the normal unprivileged user. This was because I was 
dropping privs in the wrong place, I've fixed in commit e367967, see if it 
works for you:

https://anonscm.debian.org/git/reproducible/reprotest.git/commit/?id=e367967

(The bug didn't show up because the user_group variation has some hacks in it, 
to force the fileordering variation to run as the target user.)

> [..]
> 
> What I don't understand: The second build starts with:
> 
> [..]
> INFO:root:starting build with source directory: 
> /tmp/autopkgtest.tgbi5X/build-experiment-1/, artifact pattern: ./../*.deb
> ..
> INFO:root:executing build in /tmp/autopkgtest.tgbi5X/const_build_path/
> tail: Î޷¨´ò¿ª'debian/changelog' ¶ÁȡÊý¾: ȨÏ޲»¹»
> dpkg-buildpackage: error: tail of debian/changelog subprocess returned exit 
> status 1
> INFO:root:build successful, copying artifacts
> /bin/sh: 2: cd: can't cd to /tmp/autopkgtest.tgbi5X/build-experiment-1/
> 
> but there is no /tmp/autopkgtest.tgbi5X/build-experiment-1/ directory
> anywhere:
> 
> [..]

This is explained by the fact that reprotest moves stuff from 
build-experiment-1 to const_build_path, in order to run both builds in the same 
build path. The build failed, so reprotest didn't run the cleanup to move the 
directory back. Then, reprotest thought the build was successful (it says 
"INFO:root:build successful" in your log) so it tried to do more stuff with 
/tmp/autopkgtest.tgbi5X/build-experiment-1/, but it's not there because the 
build actually failed.

The fact that reprotest thought the build was successful, was a bug in 0.7.3 
with --no-clean-on-error that I just noticed, which I accidentally fixed in a 
refactoring just before I fixed this bug. The previous code was doing

if ( run_build ); then ( cleanup ); fi

but of course this doesn't preserve the non-zero exit code from run_build, duh. 
The fix for this is in git / will be in 0.7.4 as well.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 129's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/129/

Feel free to commit fixes directly to drafts/129.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/r-base_3.4.2-1.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Question about strip-nondeterminism in bsh

[Ximin Luo]

Daniel Kahn Gillmor:
> On Thu 2017-10-12 03:02:49 +0100, Chris Lamb wrote:
> 
>> The GitHub repositories are read-only mirrors. If you are checking out
>> the code as a member of the Reproducible Builds team, then it makes more
>> sense to get it from Alioth, for example:
>>
>>   $ git clone git.debian.org:/git/reproducible/strip-nondeterminism
> 
> or:
> 
> debcheckout strip-nondeterminism
> 
> That'll give you a local mirror that you can use to track upstream
> changes, and to make your own changes.  But you won't be able to
> immediately push back to the upstream repo.
> 
> If you've made commits and you want to push them upstream and you've got
> privileges to do so on alioth (i think you do, since "jathan-guest" is a
> member of "scm_reproducible"), you can enable the local repo for pushing
> like this:
> 
>cd strip-nondeterminism
>git config remote.origin.pushurl 
> git.debian.org:/git/reproducible/strip-nondeterminism
> 
> hth,
> 

Or just `debcheckout -a [source-package-name]`. :)

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#877813: reprotest: regression between 0.7.1 and 0.7.2

[Ximin Luo]

gregor herrmann:
> [..]> dpkg-buildpackage: error: fakeroot not found, either install the 
> fakeroot
> package, specify a command with the -r option, or run this as root
> [..]
> 
> This seems to be related to the "sudo" use in the output above (or
> the funny output in the first line?), or
> 
>   * Improve the dsc+schroot preset to run builds as non-root.
> 
> in the changelog, or 62416ab in git.
> 
> After that, my python knowledge and domain knowledge ends; but
> unfortunately this means that reprotest 0.7.2 is unusable for me.
> 
Hi gregor, does it work if log into your "default" schroot and install the 
"fakeroot" package?

I am not sure why it is not installed there already - I have it installed in my 
unstable-amd64-sbuild chroot.

It would be easy enough to have reprotest install it automatically, but I'm not 
sure that's the best solution since another tool is already supposed to take 
care of that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 127's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/127/

Feel free to commit fixes directly to drafts/127.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#875282: diffoscope: AttributeError: 'NoneType' object has no attribute 'get_member'

[Ximin Luo]

Mattia Rizzolo:
> [..]
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/deb.py", line 
> 102, in control
> control_file = 
> self.as_container.control_tar.as_container.lookup_file('./control')
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/deb.py", line 
> 78, in control_tar
> return specialize(member.as_container.get_member('content'))
> AttributeError: 'NoneType' object has no attribute 'get_member'
> 

This is caused by #876316, file misdetects control.tar.gz as "DOS/MBR boot 
sector". The rest of deb.py assumes that the detection always succeeds 
correctly and does stuff like ".get_member" without checking whether it failed.

In the meantime I'll see if I can get diffoscope to work around this somehow.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876055: Environment variable handling for reproducible builds

[Ximin Luo]

Russ Allbery:
> [..]  It does mean that discovery of any new
> such environment variable would require a change to our whitelist in
> approach (3), so there would be some lag and the whitelist would become
> long over time (with a corresponding testing load).  But (3) does try to
> achieve that use case without trying to anticipate any possible
> environment variable setting.  It lets us be reactive to newly-discovered
> environment variables across which we want to stay reproducible.
> 

I can also see the merits in your (3) suggestion but I don't think it would be 
appropriate to hard-code the list in Policy, because it would be too hard to 
change it and then people might end up relying on a very-incomplete list and 
then do stupid stuff that was counter to the original intention of the 
discussions around the policy. It would be better to find a generic wording 
(with some examples) similar to what I suggested elsewhere.

>> Does everything in policy need to be rigorously testable?  or is it ok
>> to have Policy state the desired outcome even if we don't know how (or
>> don't have the resources) to test it fully today.
> 
> I don't think everything has to be rigorously testable, but I do think
> it's a useful canary.  If I can't test something, I start wondering
> whether that means I have problems with my underlying assumptions.
> 
> [..]

The "strict" interpretation is in principle testable though - we just have to 
collect enough environment variables and decide which category they fall under, 
and add that logic to our build tools.

I think in these early days, it would be fine for public package builders and 
reproducibility testers to do (3) as you suggested, i.e. 

- clean the environment
- set certain variables to a fixed value (the "whitelist") and record these in 
buildinfo

This "loose" interpretation of reproducibility still gives us some useful 
results, as well as testable reproducibility for end users, but as I said I 
don't think this should be Policy since the whitelist should be expanding quite 
quickly especially early on.

OTOH, developer reproducibility checkers (such as reprotest) can be a little 
bit more strict. I can imagine something like:

- reprotest runs 3 builds:
  - build 0 with current env
  - build 1 with current env + varying some "blacklist" envvars
  - build 2 with current env + varying some "non-whitelist" envvars

If there are differences between build 1 and build 2, then reprotest reports 
"unexpected envvar $XXX affected the build" and the developer can then either 
submit it for inclusion on the "whitelist" or the "blacklist" based on the 
Policy wording. If it ends up on the blacklist then they would also have to fix 
their own package to be invariant under that envvar.

So over time, this way we can build up a blacklist and a whitelist. But it 
shouldn't be in the original policy. And I don't think what I suggested above 
is a particularly disruptive or surprising process, especially since the 
"public" builders would only do the "looser" interpretation so people aren't 
bothered by bogus "unreproducible" reports.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#876055: Environment variable handling for reproducible builds

[Ximin Luo]

Simon McVittie:
> On Mon, 18 Sep 2017 at 18:00:51 -0700, Vagrant Cascadian wrote:
>> [..]
>>
>> I consider unintended variables that affect the build output a bug, and
>> variables designed and intended to change the behavior of the toolchain
>> expected, reasonable behavior.
> 
> There is a *huge* number of variables that are intended to change
> behaviour, and may or may not affect the behaviour of this specific
> package. Which of your categories are these in?
> 
> For example, basically any well-behaved programming language or
> programming-language-like environment has an equivalent of PYTHONPATH,
> PERL5LIB, PKG_CONFIG_PATH and similar variables, [..]
> 
> Similarly, there is an intractably huge number of environment variables
> that can affect the result of Automake and make. Do you know about all
> of them? Including RM, PC, AR, LOADLIBES (and those are just for make's
> implicit rules)? [..]
> 

I agree with this and this matches my own thoughts back in:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844431#324
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844431#369

> I think the assumption has to be that every environment variable is
> potentially intended to affect the build unless otherwise stated [..]
> [..] It would be most useful if we were to identify a
> restricted subset of environment variables for which there is consensus
> that the variable is meant to be merely user preference and shouldn't
> affect the build [..]
> 
> Perhaps those variables should be a whitelist, or perhaps there is
> some wording for Policy that would identify them while excluding the
> legitimately build-affecting ones - but either way I think the
> assumption should be "there is a limited subset of environment
> variables that are required to preserve reproducibility when varied,
> and the rest are uninteresting".
> 

These variables shouldn't be a whitelist because different buildsystems all the 
time can invent their own variables to affect themselves. We can't really 
"predict" something like PERL5LIB.

However, neither should it be a blacklist because different run-time programs 
invent their own variables all the time to affect themselves, but in a way that 
really should not affect build processes. I have to set LANG=XX.YY in my user 
environment, that doesn't mean that all my builds should run differently from 
people in other countries.

Therefore, I think it is better to try to reach some wording for Policy that 
communicates *intent*. Then, tools like dpkg-buildflag can have their own 
envvars that they force-set, which would be a subset of the ones allowed by 
Policy. Tools like reprotest can vary certain envvars that are "obviously" 
shouldn't affect the build like LC_ALL, USER, etc. Then in the middle there 
will be certain variables like RM and AR that could affect the build, which 
should be clear by Policy wording, but are too cumbersome to have 
dpkg-buildpackage try to enumerate a full whitelist and force-set them to a 
fixed value.

Interpreter variables like PER5LIB and PYTHONPATH we would have to assume fall 
in the first category ("they are allowed to affect the build output") even 
though arguably they are also "run-time variables" because they are very tied 
to the interpreter and probably only developers really want to set the for 
specific purposes.

So let's throw some wording out there already. To quote my earlier proposal:

> I would suggest amending:
> 
> - a set of environment variable values; and
> + a set of reserved environment variable values; and
> 
> then later:
> 
> + A "reserved" environment variable is defined as DEB_*, DPKG_*, 
> SOURCE_DATE_EPOCH, BUILD_PATH_PREFIX_MAP, variables listed by dpkg-buildflags 
> and other variables explicitly used by buildsystems to affect build output, 
> excluding any variables used by non-build programs to affect their behaviour. 
> Explicitly, this excludes TERM, HOME, LOGNAME, USER [..]

(The last time I erroneously included PATH in the final "excluded" list - 
because we have varied PATH but in a really trivial way on tests.r-b.org for 
ages - but I now agree with you that we shouldn't expect reproducibility when 
PATH is varied.)

My reasoning, as echoed by others on this thread already, was:

> some other variables are used by non-build tools, such as LC_*, USER, etc. 
> Since they affect non-build programs, they possibly may be set in a 
> developer's normal environment, so just running "debian/rules build" will 
> pick these up. Then, the build should stay the same despite these other 
> variables.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 125's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/125/

Feel free to commit fixes directly to drafts/125.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/gcc-7_7.2.0-8.0~reproducible1.dsc 
has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/gcc-7_7.2.0-3.0~reproducible1.dsc 
has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 123's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is now available for review:

https://reproducible.alioth.debian.org/blog/drafts/123/

Feel free to commit fixes directly to drafts/123.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: GCC build-path patch getting blocked

[Ximin Luo]

Vagrant Cascadian:
> On 2017-08-16, Ximin Luo wrote:
>> It looks like the GCC reviewer that looked at my patch this time
>> around, really doesn't like environment variables. They seem to be
>> happy to support the variable (including the syntax) as a command-line
>> flag however.
> 
>> The original patch fixed ~1800 packages, which were unreproducible due
>> to a combination of (a) __FILE__, (b) CFLAGS et al being embedded into
>> the output, and (c) packages/upstreams not honoring CFLAGS in the
>> first place, and (d) possibly other reasons.
> ...
>> For these reasons, I have the following proposal, as a work around for the 
>> time being:
>>
>> 1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this 
>> will at least fix packages affected by (a).
> 
> What about a compromise, patching GCC to support a commandline flag
> "--respect-build-path-prefix-map-variable" that tells it to respect the
> BUILD_PATH_PREFIX_MAP as an environment variable?
> 
> The commandline flag could essentially be a boolean, and would fix (a)
> as well as (b).
> 
> Or maybe GCC is just fundamentally opposed to environment variables at
> all?
> 
> 
> This is perhaps a correlary to the additional variable added to some of
> the latex toolchain that basically said "really, use SOURCE_DATE_EPOCH
> for the current time please, please, really"
> 

Thanks for the suggestion. I think it's unlikely but I'll try it and if it 
doesn't work, do what I said originally.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#844431: Revised patch: seeking seconds

[Ximin Luo]

Russ Allbery:
> Ximin Luo <infini...@debian.org> writes:
> 
>> Fair enough. I actually spotted that but thought it was better to get
>> "something" into Policy rather than nitpick. I guess other people were
>> thinking similar things. Well, lesson learnt, I will be more forceful
>> next time.
> 
>> The sentence I amended said "most environment variables" so our intent
>> is clear. If we want to fix this now, I would suggest amending:
> 
>> - a set of environment variable values; and
>> + a set of reserved environment variable values; and
> 
>> then later:
> 
>> + A "reserved" environment variable is defined as DEB_*, DPKG_, 
>> SOURCE_DATE_EPOCH, BUILD_PATH_PREFIX_MAP, variables listed by 
>> dpkg-buildflags and other variables explicitly used by buildsystems to 
>> affect build output, excluding any variables used by non-build programs to 
>> affect their behaviour. Explicitly, this excludes TERM, HOME, LOGNAME, USER, 
>> PATH and likely any variables ending with *PATH.
> 
> We intentionally didn't spell this out in this much detail because it felt
> better to defer this (stricter) bar until we have documentation of the
> *.buildinfo file, and also because we were worried about the list changing
> (once it goes into Policy, it's more irritating to change).  The current
> standard in Policy is intentionally weaker than this in order to be
> simpler.
> 
> I still lean towards taking this approach, because I'm pretty worried
> about the scope of:
> 
> other variables explicitly used by buildsystems to affect build output
> 
> That's not really an enumerable list.  My recommendation, if you want to
> allow some environment variables to vary without affecting
> reproducibility, is to explicitly list the set of environment variables
> that can vary, rather than trying to list the ones that have to remain
> fixed.
> 

Intuitively it feels weird to say "if you vary USER, the output must remain 
fixed", but also "if you vary RANDOMUNIQUESPECIALSNOWFLAKEVARIABLE then the 
output is allowed to change".

Certain environment variables have become convention to affect a build, like 
CFLAGS, and even debuild(1) doesn't clear them - but clears the other envvars. 
That is what I was going on.

> But, more fundamentally, I'm dubious that weakening the environment
> variable set is a good use of anyone's time.  Why not define reproducible
> builds as setting a specific set of environment variables and no others?
> We're long past the point where building packages in an isolated
> environment with a fixed set of environment variables is a great hardship
> or even particularly unusual.  I think the effort would be better spent on
> fixing (with enumerated exceptions) the set of environment variables set
> by buildds, sbuild, pbuilder, and other infrastructure that builds
> packages than in making packages tolerate random environment variables
> being set during the build.  It's really hard to track down all the
> environment variable settings that might affect Autoconf, the build tools,
> document formatters, and so forth.
> 

My proposal was the opposite, to *strengthen* the definition that was already 
accepted - I *don't* think we should track down all those variables and make 
packages immune to them, that is why I added "other variables explicitly used 
by buildsystems to affect build output" etc. OTOH, some other variables are 
used by non-build tools, such as LC_*, USER, etc. Since they affect non-build 
programs, they possibly may be set in a developer's normal environment, so just 
running "debian/rules build" will pick these up. Then, the build should stay 
the same despite these other variables.

If a build tool needs to be run in a specific locale, it should either use a 
locale-independent sorting program, or set LC_ALL explicitly itself regardless 
of what the parent environment says.

This doesn't contradict us from using a fixed or mostly-clean environment in 
sbuild, pbuilder, debuild, etc.

Now that I think about it however, it's probably not reasonable to expect that 
the output remains the same when PATH is changed. On tests.r-b.org we vary it 
by appending a dummy value [1] but if the user adds their own stuff to the 
beginning then the output may well change. There is probably no point in trying 
to prevent that in all packages. In a sense, it does very much affect what 
build tools are run, even though non-build programs also use it. However, my 
gut feeling still says that it's not right for the locale (LC_*) to affect a 
build process. I will try to think of a more precise way to express this 
difference.

X

[1] https://tests.reproducible-builds.org/debian/index_variations.html

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#844431: Revised patch: seeking seconds

[Ximin Luo]

Bill Allombert:
> On Tue, Aug 15, 2017 at 07:49:55PM +, Holger Levsen wrote:
>> Also what you are saying ("a package that is reproducible according to the
>> policy definition must not show up as non-reproducible in tracker/DDPO based
>> on results from the reproducible infrastructure") doesnt really makes sense:
>> if a package shows up as unreproducible somewhere, it's not reproducible
>> according to our definition!
> 
> Again, reproducible means that it _can_ be reproduced. As long as a well-know
> process allows to reproduce the package, it is reproducible.
> 
> What you define is a different concept that deserve a different name.
> 
> Cheers,
> 

I think it is OK to call this "reproducible", it's a natural language word and 
these are always dependent on some context. Technically, everything is 
reproducible if you know the state of the machine when the original build was 
started. Some other projects give you a VM and tell you to build in the VM. 
That would be a "well-known process". But nobody really knows what's in the VM 
so it's not helpful for security. Having a strict definition of 
reproducibility, helps us be more convinced that the build process is really 
only dependent on the source code and build tools, and a very restricted set of 
other inputs.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

GCC build-path patch getting blocked

[Ximin Luo]

Hi list,

It looks like the GCC reviewer that looked at my patch this time around, really 
doesn't like environment variables. They seem to be happy to support the 
variable (including the syntax) as a command-line flag however.

The original patch fixed ~1800 packages, which were unreproducible due to a 
combination of (a) __FILE__, (b) CFLAGS et al being embedded into the output, 
and (c) packages/upstreams not honoring CFLAGS in the first place, and (d) 
possibly other reasons.

Upstream are essentially arguing that fixing (c) is the proper way forward, but 
I don't think this is realistic, and unnecessarily couples two separate 
problems together - IMO reproducibility is not a CFLAGS issue, see e.g. a voice 
of support at [1]. It also doesn't fix (b); the only way to fix (b) is to add 
logic to build scripts to strip out a particular command-line flag, in the same 
way that we patched GCC to strip it out from DW_AT_producer. I don't think this 
is a clean solution either, reproducibility should come "by default" and most 
people shouldn't have to add complex logic to their own build scripts.

For these reasons, I have the following proposal, as a work around for the time 
being:

1. Patch GCC to support BUILD_PATH_PREFIX_MAP as a command-line flag. this will 
at least fix packages affected by (a).

2. Add `gcc` (et al) wrappers to strip-nondeterminism:
   /usr/share/strip-nondeterminism/bin/gcc
   /usr/share/strip-nondeterminism/bin/g++
   etc, with contents

   #!/bin/sh
   exec "$0" --path-prefix-map="${BUILD_PATH_PREFIX_MAP:-}" "$@"

3. Add a Makefile snippet to strip-nondeterminism:
   /usr/share/strip-nondeterminism/mk/build-path.mk
   with contents

   DEB_BUILD_MAINT_OPTIONS += reproducible=-fixdebugpath
   export DEB_BUILD_MAINT_OPTIONS
   PATH := /usr/share/strip-nondeterminism/bin:$(PATH)
   export PATH

Then, fixing a package affected by (b), (c) or (d) will simply consist of:

d/control:
+Build-Depends: strip-nondeterminism
d/rules:
+include /usr/share/strip-nondeterminism/mk/build-path.mk

which will be much much quicker than going in and doing more invasive patches.

This idea is similar to hardening-wrapper. That is now deprecated, but was 
useful as a stepping-stone to more "proper" fixes. Likewise, this shouldn't be 
thought of as "the proper fix", but will give us some useful data on how many 
packages are affected by which combinations of (a), (b), (c) or (d). Depending 
on the number of packages we will have to patch (with that 2-line patch), it 
will maybe give weight to future attempts to have GCC support this env var - 
and then it would be easy, since the core functionality would already be in 
there - or else highlight the issue so that maintainers of those packages fix 
things "properly".

X

[1] https://gcc.gnu.org/ml/gcc-patches/2017-08/msg00770.html 

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: More info about nondeterminism_added_by_pyqt5_pyrcc5

[Ximin Luo]

Federico Brega:
> Hello,
> 
> I'm packaging an application making use of pyrcc5 and I noticed the
> nondeterminism it adds.
> I see[1] that this is currently description is not correct.
> You can see that pyrcc5 uses QHash, which is made to avoid algorithmic
> complexity attacks[2]
> introducing a randomization.
> 
> There are two possible solutions[2]: set the environment variable
> QT_HASH_SEED to a constant value before
> pyrcc5 is called (this is my current workaround) or call 
> qSetGlobalQHashSeed().
> 
> I can help with the implementation if needed.
> 
> Regards
> --
> Federico
> 
> [1] 
> https://tests.reproducible-builds.org/debian/issues/unstable/nondeterminism_added_by_pyqt5_pyrcc5_issue.html
> [2] http://doc.qt.io/qt-5/qhash.html
> 

Hi Federico,

It might be safer to subclass QHash into a deterministic QDetHash or something. 
This would allow one to use QHash both non-deterministically (to protect 
against DoS attacks) and deterministically in the same program, depending on 
the use-case.

For example, the rust compiler internally uses a deterministic hash table but 
offers a non-deterimistic version in its standard library, see 
https://github.com/rust-lang/rust/issues/34902 for details.

You are setting seed = 0 in a header file. If this is a public header file, 
then anyone that #includes it would lose protection against those attacks, not 
just pyrcc.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#844431: Revised patch: seeking seconds

[Ximin Luo]

Sean Whitton:
> [..]
> 
> Here is an updated patch addressing these.  I reworded it to use
> 'recommended' and changed the tone to better suit policy.
> 
> Thank you Ximin, Russ and Johannes!
> 
>> "precisification" -> "more precise version"
> 
> Our definition is not actually a /version/ of the
> reproducible-builds.org definition -- that would imply that our
> definition could replace the reproducible-builds.org definition, like
> upgrading a package.
> 
> 'precisification' means roughly "filling out the missing specification
> when it is appropriate to fill it out", which is what the r-p.org
> definition instructs distributors to do.
> 
> diff --git a/policy/ch-source.rst b/policy/ch-source.rst
> index 127b125..6e32870 100644
> --- a/policy/ch-source.rst
> +++ b/policy/ch-source.rst
> @@ -661,6 +661,28 @@ particularly complex or unintuitive source layout or 
> build system (for
>  example, a package that builds the same source multiple times to
>  generate different binary packages).
>  
> +Reproducibility
> +---
> +
> +Packages should build reproducibly, which for the purposes of this
> +document [#]_ means that given
> +
> +- a version of a source package unpacked at a given path;
> +- a set of versions of installed build dependencies;
> +- a set of environment variable values;
> +- a build architecture; and
> +- a host architecture,
> +
> +repeatedly building the source package for the build architecture on
> +any machine of the host architecture with those versions of the build
> +dependencies installed and exactly those environment variable values
> +set will produce bit-for-bit identical binary packages.
> +
> +It is recommended that packages produce bit-for-bit identical binaries
> +even if most environment variables and build paths are varied.  It is
> +intended for this stricter standard to replace the above when it is
> +easier for packages to meet it.
> +
>  .. [#]
> See the file ``upgrading-checklist`` for information about policy
> which has changed between different versions of this document.
> @@ -790,3 +812,7 @@ generate different binary packages).
> often creates either static linking or shared library conflicts, and,
> most importantly, increases the difficulty of handling security
> vulnerabilities in the duplicated code.
> +
> +.. [#]
> +   This is Debian's precisification of the `reproducible-builds.org
> +   definition `_.
> 
> 

Thanks! Seconded.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/gcc-7_7.1.0-13.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

https://reproducible.alioth.debian.org/debian/gcc-7_7.1.0-12.0~reproducible1.dsc
 has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 117's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/117/

Feel free to commit fixes directly to drafts/117.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#868486: diffoscope often fails to detect APKs

[Ximin Luo]

Control: tags -1 - pending

Hans-Christoph Steiner:
> [..]
> 
> I'd like a way to force the file type in diffoscope.   We are calling it
> from a build process, so we already know all files are going to be APKs.
> Also,  I tried to get this added to libfile, but upstream is not willing
> to accept detection routines that rely on more complicated things like
> presence of a file in a ZIP. They just want byte patterns, which is not
> enough to consistently detect APKs.
> 

Do you have a link to the libfile discussion, and could you provide some more 
detail on the APK file format? I think in diffoscope we *are* happy to add more 
complicated detection logic.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

r-base_3.4.1-2.0~reproducible2.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 115's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/115/

Feel free to commit fixes directly to drafts/115.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Dealing with version.texi and mdate-sh

[Ximin Luo]

Eric Dorland:
> Hi folks,
> 
> I was trying to fix some unreproduciblity issues with automake and the
> problem of version.texi came to my attention and I haven't seen it
> come up before, but let me know if I just couldn't find it.
> 
> According to
> https://www.gnu.org/software/automake/manual/html_node/Texinfo.html,
> if your .texi includes version.texi it will generate version.texi
> based on the output of mdate-sh on the .texi file (aka whatever the
> modification date of the file is). Obviously that's bad for build
> reproducibility. 
> 
> My thinking on how to fix this would be to add a flag to mdate-sh to
> use SOURCE_DATE_EPOCH if it's available and make automake use that
> flag when generating version.texi. Is there a better approach? Should
> the unpacked source package just have all of the files modification
> dates set to SOURCE_DATE_EPOCH?
> 

Hi Eric, it's better to just directly use SOURCE_DATE_EPOCH if it's available 
without requiring an extra command-line flag, so package maintainers don't have 
to add these extra flags anywhere.

With an unpacked source package, what we generally do is reset file modtimes 
only if they are newer than SOURCE_DATE_EPOCH, otherwise keep them as-is. We've 
been calling this "clamping". If you're creating tar files you can use the 
--clamp-mtime option rather than touching the files diretcly.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.4.0-1.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

fixed

[Ximin Luo]

The current version of xz-utils built in current unstable does not seem to be 
affected by this issue. Possibly we fixed gettext in the meantime to generate 
reproducible timestamps.

It is still affected by #806331 for which I've already written a patch.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#866241: diffoscope: add a --exclude-toplevel-metadata option to help reprotest be less scary

[Ximin Luo]

Package: diffoscope
Version: 83
Severity: wishlist

Dear Maintainer,

Several times people on IRC have asked about reprotest reporting file 
permissions
differences when building files. This can be reproduced trivially:

$ reprotest 'touch x' x
[..]
│ │ │ -Access: (0644/-rw-r--r--)  Uid: ( 1000/infinity0)   Gid: ( 
1000/infinity0)
│ │ │ +Access: (0664/-rw-rw-r--)  Uid: ( 1000/infinity0)   Gid: ( 
1000/infinity0)
[..]
│ │ │ -group::r--
│ │ │ +group::rw-
[..]
exit code 1

This is because diffoscope compares the metadata of its command-line arguments
by default (which is actually a property of the parent directory and not the
file itself). For reproducibility purposes, this is not usually an issue because
one typically does not distribute x plus its current directory metadata, but
only the contents of x.

This can confuse newbies who might not be confident about what reproducibility
means, and assume that "the tool is right".

In order to avoid a diff like this, one has to do something like

$ reprotest 'umask 022; touch x' x
[..]
===
Reproduction successful
===
No differences in x
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  x

which is needlessly pointless.

However, sometimes people using diffoscope might indeed want to compare the
metadata of its arguments, e.g. after doing "make install" or similar.

Therefore, we could add a --exclude-toplevel-metadata flag to diffoscope,
which reprotest users could use via --diffoscope-arg, and this could also
be set by default in any relevant "presets".

This would cover both use cases.

X


-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 
'testing-debug'), (300, 'unstable'), (200, 'experimental'), (1, 
'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8), LANGUAGE=en_GB:en 
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python33.5.3-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  1:5.30-1
ii  python3-pkg-resources  33.1.1-1

Versions of packages diffoscope recommends:
ii  acl  2.2.52-3+b1
ii  apktool  2.2.1+dfsg-2
ii  binutils-multiarch   2.28-5
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b2
ii  colord   1.3.3-2
ii  default-jdk [java-sdk]   2:1.8-58
ii  default-jdk-headless 2:1.8-58
ii  enjarify 1:1.0.3-3
ii  fontforge-extras 0.3-4
ii  fp-utils 3.0.0+dfsg-11
ii  fp-utils-3.0.0 [fp-utils]3.0.0+dfsg-11
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-2
ii  ghc  8.0.1-17+b1
ii  ghostscript  9.20~dfsg-3.2
ii  gnupg2.1.18-6
ii  imagemagick  8:6.9.7.4+dfsg-11
ii  imagemagick-6.q16 [imagemagick]  8:6.9.7.4+dfsg-11
ii  jsbeautifier 1.6.4-6
ii  llvm 1:3.8-36
ii  mono-utils   4.6.2.7+dfsg-1
ii  openjdk-8-jdk [java-sdk] 8u131-b11-2
ii  openssh-client   1:7.4p1-10
ii  pdftk2.02-4+b2
ii  poppler-utils0.48.0-2
ii  python3-argcomplete  1.8.1-1
ii  python3-debian   0.1.30
ii  python3-guestfs  1:1.34.6-2
ii  python3-progressbar  2.3-4
ii  python3-rpm  4.12.0.2+dfsg1-2
ii  python3-tlsh 3.4.4+20151206-1+b2
ii  rpm2cpio 4.12.0.2+dfsg1-2
ii  sng  1.1.0-1+b1
ii  sqlite3  3.16.2-5
ii  squashfs-tools   1:4.3-3+b1
ii  unzip6.0-21
ii  vim-common   2:8.0.0197-4
ii  xxd  2:8.0.0197-4
ii  xz-utils 5.2.2-1.2+b1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.1.1-2

-- no debconf information
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

r-base_3.4.0.20170622-1.0~reproducible2.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 113's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/113/

Feel free to commit fixes directly to drafts/113.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: source-only builds and .buildinfo

[Ximin Luo]

Adrian Bunk:
> On Thu, Jun 22, 2017 at 08:26:00AM +0000, Ximin Luo wrote:
>> ...
>> One way to give security that is independent of third parties, is to provide 
>> some sort of mathematically-verifiable proof. However the world isn't at 
>> that stage yet for compiler technology.
> 
> What changes in compiler technology are you hoping for?
> 
> The main reason for fixing optimizer bugs in the compiler is to get 
> different (no longer buggy) output.
> 
>> ...
>> For users that can't directly verify everything that they themselves run, 
>> one "next best thing" they can do is to check that different parties that 
>> they trust - or many parties that they don't trust, that they nevertheless 
>> believe are probably not all colluding to attack them - claimed to have 
>> performed the build or verified each others' proofs.
>>
>> So, the more buildinfo files we have, from different parties (DDs, the 
>> Debian archive, etc) the better this is for users, because they have more 
>> sources of claims. How much they "trust" each individual source, is indeed 
>> not something that is concretely measurable and no existing security system 
>> tries to model this more precisely unfortunately; however I think we can all 
>> agree that "more is better" here.
>> ...
> 
> I don't see how more random information is helpful for users.
> 
> One or more trusted instances verifying that all packages in a release 
> were built from their sources is the information that would be useful 
> for users.
> 

Different users can choose who they want to trust. A DD signing a buildinfo and 
uploading this to the archive, is not "random information". Some users would be 
happy to trust 1 DD plus the buildd, but not either one individually; other 
users would want other third-party builders to re-perform the build and sign 
their own buildinfo files.

The point is that making the information available gives more choice for users. 
If specific users don't trust a DD, they can ignore this extra information. But 
if we don't provide this information, we're preventing people from getting 
assurance about the software they're running.

BTW this sort of trust-system I'm suggesting is not like the CA system where 1 
trusted party can break your security. Instead, here all/most trusted parties 
would have to collude to publish bad buildinfo files, to break your security. 
The security dynamics, is closer to bitcoin and other blockchain tech. There 
are certain nuances to be made when doing the security logic, for example 
someone could sign 100 bad buildinfo files pretending to be from different 
people; but I think the success of blockchain tech shows that there is some 
demand from users to have these AND-trust systems where many trusted sources, 
even from "random strangers", can help make the system stronger, as opposed to 
OR-trust systems where 1 CA can go MITM everyone.

> For some users it would also be important to be able to verify this for 
> the whole archive themselves.
> 

Yeah, we agree on this point. For many other users, who don't have the 
resources to rebuild everything, it's equally important to be able to see that 
{whatever they choose} other people have claimed to have done it.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: source-only builds and .buildinfo

[Ximin Luo]

Adrian Bunk:
> On Wed, Jun 21, 2017 at 10:09:00AM +0000, Ximin Luo wrote:
>> Adrian Bunk:
>>> [..]
>>>
>>> How is that supposed to work when the compiler is not exactly identical?
>>>
>>> As an example, gcc-6 6.3.0-18 and gcc-6 6.3.0-19 will likely produce 
>>> different output for every non-trivial piece of software.
>>>
>>> The reason is that every new gcc upload usually contains whatever 
>>> bugfixes are on the upstream branch.
>>>
>>
>> It would depend on the situation which dependencies should be "irrelevant" 
>> towards the final output, right. If the dependencies are different and the 
>> buildinfo is different, it does not necessarily mean there is a problem, the 
>> upload does not need to be rejected. But it's a signal that other people 
>> (including the uploader) might want to re-try the build with the newer 
>> dependencies.
>>
>> OTOH if the outputs match, we get more certainty, which is a good thing.
>> ...
> 
> "more certainty" on what exactly?
> 

More certainty that the binaries produced, were actually produced from the 
source code, rather than by malicious or compromised machines.

> "signal that other people might want to" is quite vague,
> what do you want to prove and how exactly should people
> spend time proving it?
> 

That the binaries uploaded were actually produced from the source code. People 
spend time proving it by running the build against and seeing if the binaries 
match, possibly also recreating various aspects of previous build environments 
recorded in other .buildinfo files.

> In the best case [excluding the binary-all special case] we would know that 
> the buildd on the one 
> architecture that happens to be used by the person doing the
> source upload produced the same binaries.
> 
> Once you start verifying that all binaries in the archive were built 
> from the sources in the archive, this will automatically be covered.
> 

What we'd like to aim for, is to give users some security guarantee 
*independent of the distributor i.e. Debian or DD uploaders* that the binaries 
they're using is actually produced from the source code.

One way to give security that is independent of third parties, is to provide 
some sort of mathematically-verifiable proof. However the world isn't at that 
stage yet for compiler technology.

Buildinfo files are more like claims rather than proofs. Whilst it can be used 
as a proof, i.e. by running the build yourself, this is an expensive process 
which we can't expect most users to do, and doesn't really fit the idea of a 
"proof" in a security system, which are supposed to be low-cost for verifiers.

For users that can't directly verify everything that they themselves run, one 
"next best thing" they can do is to check that different parties that they 
trust - or many parties that they don't trust, that they nevertheless believe 
are probably not all colluding to attack them - claimed to have performed the 
build or verified each others' proofs.

So, the more buildinfo files we have, from different parties (DDs, the Debian 
archive, etc) the better this is for users, because they have more sources of 
claims. How much they "trust" each individual source, is indeed not something 
that is concretely measurable and no existing security system tries to model 
this more precisely unfortunately; however I think we can all agree that "more 
is better" here.

Therefore, there is still value in using DDs' uploaded buildinfo files, even if 
the buildds are "likely" to use different dependencies and "likely" to produce 
different binaries. If they have identical output, great, they get a nice green 
tick somewhere. If not, people can run the builds again to try to get the 
identical output. And some builds are indeed not reproducible today, and these 
indicate bugs rather than builders being compromised.

Besides, I think "non-identical builds due to changed dependencies" won't 
actually be so likely in practice. For example GCC-6 -18 was there for 3-4 
weeks, and plenty of uploads happened during that time. Most DDs would update, 
build and upload within several minutes or hours of each other.

X


-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: source-only builds and .buildinfo

[Ximin Luo]

Adrian Bunk:
> On Wed, Jun 21, 2017 at 09:28:00AM +0000, Ximin Luo wrote:
>> Adrian Bunk:
>>> On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote:
>>>> Hi Ian--
>>>>
>>>> On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote:
>>>>> A .buildinfo file is not useful for a source-only upload which is
>>>>> veried to be identical to the intended source as present in the
>>>>> uploader's version control (eg, by the use of dgit).
>>>>>
>>>>> Therefore, dgit should not include .buildinfos in source-only uploads
>>>>> it performs.  If dgit sees that a lower-layer tool like
>>>>> dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit
>>>>> should strip it out of .changes.
>>>>
>>>> I often do source-only uploads which include the .buildinfo.
>>>>
>>>> I do source-only uploads because i don't want the binaries built on my
>>>> own personal infrastructure to reach the public.  But i want to upload
>>>> the .buildinfo because i want to provide a corroboration of what i
>>>> *expect* the buildds to produce.
>>>> ...
>>>
>>> If you expect that, then your expectation is incorrect.
>>>
>>> If you upload a package right now, chances are the buildds will use both 
>>> older versions of some packages [1] and more recent versions of some 
>>> other packages [2] than what you used.
>>>
>>
>> I think what dkg means here (and what we the R-B team has wanted for ages 
>> and is working towards), is not that the buildds use the *versioned 
>> dependencies* listed in the buildinfo, but produce the same *output hashes* 
>> as what's in the buildinfo.
>>
>> The point being specifically that the dependencies used could change, but if 
>> the output remains constant, we're more assured that the build was done 
>> properly and reproducibly.
> 
> How is that supposed to work when the compiler is not exactly identical?
> 
> As an example, gcc-6 6.3.0-18 and gcc-6 6.3.0-19 will likely produce 
> different output for every non-trivial piece of software.
> 
> The reason is that every new gcc upload usually contains whatever 
> bugfixes are on the upstream branch.
> 

It would depend on the situation which dependencies should be "irrelevant" 
towards the final output, right. If the dependencies are different and the 
buildinfo is different, it does not necessarily mean there is a problem, the 
upload does not need to be rejected. But it's a signal that other people 
(including the uploader) might want to re-try the build with the newer 
dependencies.

OTOH if the outputs match, we get more certainty, which is a good thing.

We also need to get some real data on this, it could be that a change from -18 
to -19 would only affect a small number of packages, and most other ones would 
actually be compiled identically.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-19.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 111's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/111/

Feel free to commit fixes directly to drafts/111.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#863636: diffoscope: usage of FIFOs causes pair-comparisons to not run in parallel, wasting performance by about 1/2

[Ximin Luo]

Ximin Luo:
> Ximin Luo:
>> Package: diffoscope
>> Version: 78
>> Severity: normal
>>
>> Dear Maintainer,
>>
>> diff(1) first reads the contents of one file then the next one:
>>
>> https://sources.debian.net/src/diffutils/1:3.5-3/src/io.c/#L552
>>
>> This means that if the "files" are actually FIFOs connected to the output of 
>> a
>> process, as they are in many cases in diffoscope, the second process has to 
>> wait
>> for diff(1) to fully read the output of the first process, before it itself 
>> can
>> run. This prevents both processes from running in parallel.
>>
>> An appropriate fix would be to store the output of at least one of the 
>> commands
>> into a temporary file, and have diff(1) read from this instead. This has to 
>> be
>> done carefully however, to make sure that diff(1) doesn't accidentally read 
>> it
>> before the process is finished.
>>
>> [..]
> 
> It seems readelf specifically has weird performance behaviours when running 
> in parallel.
> 
> [..]

I couldn't reproduce the above results on Holger's profitbricks machine, and 
bunk@ couldn't reproduce it either. That is, running the commands in parallel 
*did* produce roughly a 2x speed up.

Also on my local machine I got:

$ ls -laSr /usr/bin/{hot,hokey,darcs}
-rwxr-xr-x 1 root root 20555008 Oct 28  2016 /usr/bin/hot*
-rwxr-xr-x 1 root root 29637664 Oct 28  2016 /usr/bin/hokey*
-rwxr-xr-x 1 root root 37144392 Oct 28  2016 /usr/bin/darcs*

$ f() { taskset --cpu-list $1 objdump -S /usr/bin/hot >/dev/null; }; time ( 
f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m12.445s
user0m12.408s
sys 0m0.024s

real0m7.653s
user0m15.224s
sys 0m0.040s

$ f() { taskset --cpu-list $1 objdump -S /usr/bin/hokey >/dev/null; }; time 
( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m24.998s
user0m24.896s
sys 0m0.064s

real0m21.197s
user0m42.224s
sys 0m0.076s

$ f() { taskset --cpu-list $1 objdump -S /usr/bin/darcs >/dev/null; }; time 
( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m38.652s
user0m38.532s
sys 0m0.064s

real0m34.323s
user1m8.168s
sys 0m0.104s

i.e. the speed-improvement-due-to-parallelism decreases as the size of the 
input increases - but I couldn't reproduce this the profitbricks machine either.

Due to the lack of debugging symbols for binutils (#863728) it's hard for me to 
investigate this further, so I'll pause this for now.

It's probably worth un-reverting e28b540b0b289ce9fda70095160382799d7602a6 
perhaps guarded by a CLI flag; though diffoscope's heavy use of Python-based 
filtering of external commands' output makes this less significant (without 
also trying to optimise how this filtering is done). In the meantime I'm also 
using "--exclude-command '^readelf.*\s--debug-dump=info'" to avoid the longest 
part of ELF processing.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#863636: diffoscope: usage of FIFOs causes pair-comparisons to not run in parallel, wasting performance by about 1/2

[Ximin Luo]

Ximin Luo:
> Package: diffoscope
> Version: 78
> Severity: normal
> 
> Dear Maintainer,
> 
> diff(1) first reads the contents of one file then the next one:
> 
> https://sources.debian.net/src/diffutils/1:3.5-3/src/io.c/#L552
> 
> This means that if the "files" are actually FIFOs connected to the output of a
> process, as they are in many cases in diffoscope, the second process has to 
> wait
> for diff(1) to fully read the output of the first process, before it itself 
> can
> run. This prevents both processes from running in parallel.
> 
> An appropriate fix would be to store the output of at least one of the 
> commands
> into a temporary file, and have diff(1) read from this instead. This has to be
> done carefully however, to make sure that diff(1) doesn't accidentally read it
> before the process is finished.
> 
> [..]

It seems readelf specifically has weird performance behaviours when running in 
parallel. My initial attempt at fixing this works in the general case, but not 
for specific cases that are important to diffoscope and this scenario.

https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?h=experimental=e28b540b0b289ce9fda70095160382799d7602a6

Taking diffoscope completely out of the equation, we see the following 
behaviours:

$ f() { sleep 3; }; time ( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m6.010s
user0m0.000s
sys 0m0.000s

real0m3.004s
user0m0.000s
sys 0m0.000s

$ f() { sha256sum /run/shm/elf.$1 >/dev/null; }; time ( f 1; f 2; ); time ( 
f 1 & x=$!; f 2; wait $x; )

real0m0.451s
user0m0.432s
sys 0m0.012s

real0m0.221s
user0m0.428s
sys 0m0.004s

(The above two cases confirm that the shell snippets are correct. 
/run/shm/elf.{1,2} is about 50MB.)

$ f() { taskset $1 readelf --wide --debug-dump=rawline /run/shm/elf.$1 
>/dev/null; }; time ( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m17.128s
user0m17.096s
sys 0m0.008s

real0m19.126s
user0m37.256s
sys 0m0.016s

$ f() { taskset $1 readelf --wide --debug-dump=rawline 
/usr/lib/debug/.build-id/22/46ba050897f1d98034a7ca4b7ec06b594a373d.debug 
>/dev/null; }; time ( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m3.330s
user0m3.324s
sys 0m0.000s

real0m1.674s
user0m3.324s
sys 0m0.000s

$ f() { taskset $1 readelf --wide --debug-dump=info 
/usr/lib/debug/.build-id/22/46ba050897f1d98034a7ca4b7ec06b594a373d.debug 
>/dev/null; }; time ( f 1; f 2; ); time ( f 1 & x=$!; f 2; wait $x; )

real0m18.437s
user0m18.384s
sys 0m0.012s

real0m29.039s
user0m56.196s
sys 0m0.044s

/usr/lib/debug/.build-id/22/46ba050897f1d98034a7ca4b7ec06b594a373d.debug is 
about 4MB and from libc6-dbg=2.24-10.

So it looks like readelf has issues when running for a long time in parallel. 
(--debug-dump=rawline with a 4MB file sped up when running in parallel.)

However other tools don't seem to have this problem:

$ time ( { echo 1; echo 2; } | xargs -n1 -I '{}' taskset '{}' readelf 
--wide --debug-dump=info 
/usr/lib/debug/.build-id/22/46ba050897f1d98034a7ca4b7ec06b594a373d.debug 
>/dev/null )

real0m18.730s
user0m18.644s
sys 0m0.032s

$ time ( { echo 1; echo 2; } | parallel --will-cite taskset '{}' readelf 
--wide --debug-dump=info 
/usr/lib/debug/.build-id/22/46ba050897f1d98034a7ca4b7ec06b594a373d.debug 
>/dev/null )

real0m33.076s
user1m0.844s
sys 0m0.460s

$ time ( { echo 1; echo 2; } | xargs -n1 -I '{}' taskset '{}' sha256sum 
/run/shm/test.iso >/dev/null )

real0m33.302s
user0m32.332s
sys 0m0.932s

$ time ( { echo 1; echo 2; } | parallel --will-cite taskset '{}' sha256sum 
/run/shm/test.iso >/dev/null )

real0m16.843s
user0m32.224s
sys 0m1.272s

Here, /run/shm/test.iso is about 4GB in size, and it does speed up when being 
processed by sha256sum in parallel.

Running strace on the "readelf" calls shows that most syscalls are done at the 
start, then for most of the rest of the execution it's just "write()" syscalls 
and indeed the "sys" times above are negligible. So it seems unlikely that the 
processes are blocking on anything.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#863636: diffoscope: usage of FIFOs causes pair-comparisons to not run in parallel, wasting performance by about 1/2

[Ximin Luo]

Package: diffoscope
Version: 78
Severity: normal

Dear Maintainer,

diff(1) first reads the contents of one file then the next one:

https://sources.debian.net/src/diffutils/1:3.5-3/src/io.c/#L552

This means that if the "files" are actually FIFOs connected to the output of a
process, as they are in many cases in diffoscope, the second process has to wait
for diff(1) to fully read the output of the first process, before it itself can
run. This prevents both processes from running in parallel.

An appropriate fix would be to store the output of at least one of the commands
into a temporary file, and have diff(1) read from this instead. This has to be
done carefully however, to make sure that diff(1) doesn't accidentally read it
before the process is finished.

X

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 
'testing-debug'), (500, 'buildd-unstable'), (300, 'unstable'), (100, 
'experimental'), (1, 'experimental-debug')
Architecture: amd64
 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic  1:5.30-1
ii  python3-pkg-resources  33.1.1-1
pn  python3:any

Versions of packages diffoscope recommends:
ii  acl  2.2.52-3+b1
ii  apktool  2.2.1+dfsg-2
ii  binutils-multiarch   2.28-5
ii  bzip21.0.6-8.1
ii  caca-utils   0.99.beta19-2+b2
ii  colord   1.3.3-2
ii  default-jdk [java-sdk]   2:1.8-58
ii  default-jdk-headless 2:1.8-58
ii  enjarify 1:1.0.3-3
ii  fontforge-extras 0.3-4
pn  fp-utils 
ii  genisoimage  9:1.1.11-3+b2
ii  gettext  0.19.8.1-2
ii  ghc  8.0.1-17+b1
ii  ghostscript  9.20~dfsg-3.2
ii  gnupg2.1.18-6
ii  imagemagick  8:6.9.7.4+dfsg-8
ii  imagemagick-6.q16 [imagemagick]  8:6.9.7.4+dfsg-8
ii  jsbeautifier 1.6.4-6
ii  llvm 1:3.8-36
ii  mono-utils   4.6.2.7+dfsg-1
ii  openjdk-8-jdk [java-sdk] 8u131-b11-2
ii  openssh-client   1:7.4p1-10
ii  pdftk2.02-4+b2
ii  poppler-utils0.48.0-2
ii  python3-argcomplete  1.8.1-1
ii  python3-debian   0.1.30
pn  python3-guestfs  
ii  python3-progressbar  2.3-4
ii  python3-rpm  4.12.0.2+dfsg1-2
ii  python3-tlsh 3.4.4+20151206-1+b2
ii  rpm2cpio 4.12.0.2+dfsg1-2
ii  sng  1.1.0-1+b1
ii  sqlite3  3.16.2-3
ii  squashfs-tools   1:4.3-3+b1
ii  unzip6.0-21
ii  vim-common   2:8.0.0197-4
ii  xxd  2:8.0.0197-4
ii  xz-utils 5.2.2-1.2+b1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.1.1-2

-- no debconf information

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: source-only builds and .buildinfo

[Ximin Luo]

Ian Jackson:
> [..]
> 
>   
> https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_are_.buildinfo_files_always_generated_with_dpkg-buildpackage.3F
>  
> 
> As I wrote to Guillem, quoting the FAQ:
> 
>>   By default dpkg-buildpackage does active tasks such as cleaning via
>>   debian/rules, and makes sure that the dependencies from Build-Depends
>>   are satisfied as these are needed by the clean target. In addition the
>>   clean target can perform any kind of action that will affect the
>>   source package, which is also part of the build, and should be by
>>   default also reproducible
>>
>> I think what you mean here is that one might have a source package
>> which is not a fixed point under `debian/rules clean' for all
>> reasonable combinations of the build-deps.  I think this is a buggy
>> package but in practice it seems that many packages are buggy in this
>> way.
>>
>> Indeed IMO it is a defect of our overall design that it the concept of
>> a `non-reproducible source package' even exists.  Sources are the
>> input to builds, not an output, so the question of reproducing them
>> does not arise.  That our system as a whole can sometimes mutate the
>> source package all by itself is a bug.
>>

I actually would like to see this fixed, then we can put source and binary 
hashes in /var/lib/dpkg/status for every binary package, then we can add these 
to .buildinfo files, which is more secure than adding the version number (as we 
do now).

I agree this is a separate issue, but I have some concrete suggestions that I 
could go into in another thread, if anyone is interested.

Also the man page for dpkg-buildpackage is out-of-date:

   6. Unless a source-only build has been requested, it runs the buildinfo 
hook and calls dpkg-genbuildinfo to generate a .buildinfo file.  Several 
dpkg-buildpackage options are forwarded to dpkg-genbuildinfo.

and also later:

  The current hook-name supported are:
  init preclean source build binary changes postclean check sign 
done

missing out "buildinfo", and indeed if I run "dpkg-buildpackage 
--hook-buildinfo=true" the buildinfo file still gets generated.

>> However, these are not considerations for dgit in this context, since
>> what dgit uploads is always guaranteed to be equal to the input.
>> Often the user will have dgit use `git clean' rather than rules clean;
>> and even if they don't, dgit will check that the results were the
>> same.
>>
>> That is, even with the .buildinfo, someone who gets the .dsc cannot
>> know whether the rules clean target is correct (or to put it another
>> way, under what conditions the source tree is a fixed point under
>> rules clean), because dgit has not necessarily run rules clean at all.
>> I'm sure there are other vcsish tools which have the same property.
>>
>> (Also, and I hesitate to make this argument because of course I
>> support reproducible builds, but: if the .buildinfo is not useful,
>> then it's an unwarranted privacy violation.)
>>
>> So I think for `dgit push-source', there should be no .buildinfo ?
>> At least, unless dgit ran the clean target.
>>
>> This suggests to me that dgit push-source should use dpkg-source
>> rather than dpkg-buildpackage, as you note in later in the FAQ entry:
>>
>>   If the intention is to just produce a source package instead of an
>>   actual build to upload, then using dpkg-source is always the better
>>   option.
>>
>> This wording is a bit unclear.  It conflates `build' and `for upload'.
>> I think regarding `dgit push-source' as a build is perverse.
>>
>> dgit would have to run dpkg-genchanges.
>>
>> Alternatively dgit could strip out the .buildinfo, depending on
>> whether it ran rules clean.
> 
> What do you think ?
>
> (The background here is that `dgit push-source' wants to verify for
> itself that the .changes file it is uploading is really source-only.
> Because of the possible presence of extraneous (eg BY-HAND) build
> artefacts in .changes, Guillem suggested comparing the .changes to the
> .dsc.  But of course the .changes contains not only the .dsc and the
> files named in it, but also the .buildinfo.)
> 

There are a few other options for you:

- Add a --no-buildinfo flag to dpkg-genchanges, then call dpkg-buildpackage 
--changes-option=--no-buildinfo
- Ignore the buildinfo entry in the .changes file.
- Verify that the buildinfo file contains only ".dsc" entries and that they 
match up with the ones in the changes file.

I'm actually not sure what your main problem is. Does dgit by default checkout 
a previously-build .dsc from git? And you are worried that if 
"dpkg-buildpackage -S" is run, causing "debian/rules clean" to be run, that the 
second-built .dsc would differ from the one that is checked in?

If this is the case, you have this problem regardless of whether the .changes 
file contains a .buildinfo file or not, these are two separate issues.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE

package uploaded to our repo

[Ximin Luo]

dpkg_1.18.24.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-18.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: #862112: r-base-dev: Generate reproducible output independently of the build path

[Ximin Luo]

Holger Levsen:
> control: found -1 3.4.0-1
> control: notfound -1 3.4.0-1.0~reproducible2
> 
> Hi Dirk,
> 
> I've asked Ximin to file this bug so that we have something to track and to
> refer in discussions…
> 
> you wrote:
> 
>> At work now with limited time, but I think I already told you twice that
>> there were two of the three parts of the patch you proposed to upstream that
>> I would not take, were I upstream (which I am not).
> 
> Dirk, could you please point out (here in this bug report) which of the two 
> parts
> you consider unsuitable?
> 

We talked about this over private email, this refers to the patch hunks in:

- src/library/tools/R/admin.R
- src/library/tools/R/parseRd.R

The suggestion was to guard them behind a command-line option using getOption, 
so presumably Debian could set this whilst upstream / other distros do not.

However, since I'm not familiar with the full R codebase I was waiting to see 
if upstream had any further comments, because even with the getOption() change 
there might be other consequences that I didn't foresee. In this case it would 
be beneficial for Debian's behaviour to exactly match upstream, so we get the 
same bugs (if any).

> Ximin, looking at 
> https://anonscm.debian.org/cgit/reproducible/r-base.git/log/?h=pu/reproducible_builds
> I believe it would be best to merge those two (top most) commits into one?
> 
>> A decent longer-term strategy may well to stress-test the patch by including
>> it in our builds for a while.  We can work on that.
> 
> actually we've been using Ximin's patches on tests.reproducible-builds.org
> since the 2nd of May on amd64 and since the 3rd on arm64, armhf and i386.
> 
> According to 
> https://tests.reproducible-builds.org/debian/index_performance.html
> (top row labeled "oldest build result) in the first table on the right) this
> means we've almost done a full rebuild with the patch on amd64+arm64 (probably
> 85% done now) and pretty close to that on i386…
> 
> According to this, this patch seems to work well…
> 

The patch works well for getting stuff built, but I haven't tested all of the 
*behaviour* of these packages. (Some probably have unit tests, but these don't 
cover everything at any rate.) That is why I wanted to wait for upstreams' 
comments first, before adding the getOption guards - which are relatively minor 
IMO, compared to what the full patch does.

I also haven't tested other potential tools that might process R rdb files, who 
might for some reason expect absolute build paths.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-17.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [rb-general] Unreproducible test logs in output

[Ximin Luo]

Chris Lamb:
> Ximin Luo wrote:
> 
>> How shall we handle this in the context of reproducible builds?
> 
> Devil's advocate for a second - it is sufficient that the output
> appears on https://buildd.debian.org so anyone interested enough can
> look there.
> 
> Whilst it would be missing for the maintainer's "own" architecture
> right now, source-only uploads — hopefully introduced during stretch's
> lifetime — would fix that hole.
> 

The retrieval process is more fiddly and it's not standardised across different 
deb-based distributions. You might also want the output to be more structured 
than just one long build log - for example binutils splits up the results by 
architecture.

The maintainer clearly thought it was more convenient to bundle the test 
results in the binary package, than to look them up on the buildd web interface 
- so just saying "don't do it" isn't a very user-friendly approach.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Unreproducible test logs in output

[Ximin Luo]

I meant to send this to rb-general as well:

Ximin Luo:
> I've been reviewing Debian's "required" and "build-essential" package sets. 
> One new issue that's come up, is that the gcc-6 and binutils packages both 
> put test logs in their binary output.
> 
> Of course this has no chance of ever being reproducible, due to parallelism, 
> build paths, and many many things. I don't think it would be reasonable to 
> try to work towards making this reproducible.
> 
> The justification for *including* them is that many very very large programs 
> have test suites that aren't expected to completely pass all the time. Now, 
> sometimes it's possible to ignore specific failing tests, and that is what I 
> have been personally doing for the rust compiler in Debian. 
> 
> However, in other cases there are just too many, and the maintainer(s) 
> physically don't have enough time to ignore these tests, since nothing else 
> useful will get done. I guess this is the case for gcc-6 and binutils, and I 
> know this is the case for another very large program I help maintain in 
> Debian, sagemath with ~300k test cases.
> 
> In these cases, it is much easier for the package maintainer to run the 
> tests, pass the build even if their are failures, and distribute the test 
> logs along with the output so that they can be examined later. Indeed I've 
> been very tempted to do this myself for sagemath.
> 
> How shall we handle this in the context of reproducible builds?
> 
> As a first proposal, perhaps we could allow package maintainers to define a 
> set of "non-installable binary output" that does not have to be reproducible. 
> This would only be used in very exceptional circumstances such as the ones 
> described above, they can't be installed via the package manager, but they 
> can be downloaded and manually extracted to analyse or distribute test 
> results with.
> 
> OTOH if we tell package maintainers "don't put test logs in binary output", 
> we need to offer an alternative that caters to the needs of very very large 
> packages, as described earlier. A simple interface to download the build logs 
> of a given binary package, might be sufficient.
> 
> X
> 

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Unreproducible test logs in output

[Ximin Luo]

I've been reviewing Debian's "required" and "build-essential" package sets. One 
new issue that's come up, is that the gcc-6 and binutils packages both put test 
logs in their binary output.

Of course this has no chance of ever being reproducible, due to parallelism, 
build paths, and many many things. I don't think it would be reasonable to try 
to work towards making this reproducible.

The justification for *including* them is that many very very large programs 
have test suites that aren't expected to completely pass all the time. Now, 
sometimes it's possible to ignore specific failing tests, and that is what I 
have been personally doing for the rust compiler in Debian. 

However, in other cases there are just too many, and the maintainer(s) 
physically don't have enough time to ignore these tests, since nothing else 
useful will get done. I guess this is the case for gcc-6 and binutils, and I 
know this is the case for another very large program I help maintain in Debian, 
sagemath with ~300k test cases.

In these cases, it is much easier for the package maintainer to run the tests, 
pass the build even if their are failures, and distribute the test logs along 
with the output so that they can be examined later. Indeed I've been very 
tempted to do this myself for sagemath.

How shall we handle this in the context of reproducible builds?

As a first proposal, perhaps we could allow package maintainers to define a set 
of "non-installable binary output" that does not have to be reproducible. This 
would only be used in very exceptional circumstances such as the ones described 
above, they can't be installed via the package manager, but they can be 
downloaded and manually extracted to analyse or distribute test results with.

OTOH if we tell package maintainers "don't put test logs in binary output", we 
need to offer an alternative that caters to the needs of very very large 
packages, as described earlier. A simple interface to download the build logs 
of a given binary package, might be sufficient.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

r-base_3.4.0-1.0~reproducible2.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 105's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/105/

Feel free to commit fixes directly to drafts/105.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-16.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

r-base_3.4.0-1.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 103's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/103/

Feel free to commit fixes directly to drafts/103.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-14.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

dpkg_1.18.23.0~reproducible2.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-12.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: dpkg and gcc-6 uploaded with BUILD_PATH_PREFIX_MAP support

[Ximin Luo]

Ximin Luo:
> Hi all, I uploaded patched versions of dpkg and gcc-6 today.
> 
> "Soon" (after all the schroots are updated) we should see how well this 
> works. Hopefully we'll get to around 88-90% or so reproducibility again in 
> sid; there are still some issues like [1] that aren't covered by these 
> patches.
> 
> The patches are here:
> 
> https://anonscm.debian.org/cgit/reproducible/debrepatch.git/tree/toolchain-patches
> 
> or as git commits here:
> 
> https://anonscm.debian.org/cgit/reproducible/gcc-6.git/
> https://anonscm.debian.org/cgit/reproducible/dpkg.git/
> 
> I haven't yet filed bugs for them because they are not complete - e.g. they 
> need documentation and upstream approval. But I'll do this soon in the next 
> few weeks or so.
> 
> X
> 
> [1] 
> https://tests.reproducible-builds.org/debian/issues/unstable/randomness_in_r_rdb_rds_databases_issue.html
> 

Looks like gcc-6 got updated in the meantime, invalidating the changes - and 
making some more things unreproducible because I had disabled the 
-fdebug-prefix-map fix in dpkg.

I've removed these patches now. On Monday I'll reupload dpkg but only disable 
-fdebug-prefix-map on amd64, and then try to keep gcc-6 (amd64 only) updated 
over the next few weeks. It seems doko is uploading new versions roughly once 
every 1-2 weeks, and if I use pb17 to build with DEB_BUILD_OPTIONS=nocheck the 
build should only take 1-2 hours.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: dpkg and gcc-6 uploaded with BUILD_PATH_PREFIX_MAP support

[Ximin Luo]

Ximin Luo:
> Chris Lamb:
>> Ximin Luo wrote:
>>
>>> Hi all, I uploaded patched versions of dpkg and gcc-6 today.
>>>
>>> "Soon" (after all the schroots are updated) we should see how well
>>> this works
>>
>> Similar to the i386 issue yesterday, are we missing an armhf build? I'm
>> seeing a lot of packages becoming unreproducible there that were
>> previously fine, eg:
>>
>>   https://tests.reproducible-builds.org/debian/unstable/armhf/numlockx
>>
> 
> Yes, gcc-6 arm64 and armhf are still missing. I'm having trouble building 
> arm64 on asachi, got the following error twice:
> 
> https://paste.debian.net/926227/
> 

Looks like `dpkg-buildpackage -jauto` screws with more things than I expected, 
retrying the build without it.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: dpkg and gcc-6 uploaded with BUILD_PATH_PREFIX_MAP support

[Ximin Luo]

Chris Lamb:
> Ximin Luo wrote:
> 
>> Hi all, I uploaded patched versions of dpkg and gcc-6 today.
>>
>> "Soon" (after all the schroots are updated) we should see how well
>> this works
> 
> Similar to the i386 issue yesterday, are we missing an armhf build? I'm
> seeing a lot of packages becoming unreproducible there that were
> previously fine, eg:
> 
>   https://tests.reproducible-builds.org/debian/unstable/armhf/numlockx
> 

Yes, gcc-6 arm64 and armhf are still missing. I'm having trouble building arm64 
on asachi, got the following error twice:

https://paste.debian.net/926227/

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

dpkg and gcc-6 uploaded with BUILD_PATH_PREFIX_MAP support

[Ximin Luo]

Hi all, I uploaded patched versions of dpkg and gcc-6 today.

"Soon" (after all the schroots are updated) we should see how well this works. 
Hopefully we'll get to around 88-90% or so reproducibility again in sid; there 
are still some issues like [1] that aren't covered by these patches.

The patches are here:

https://anonscm.debian.org/cgit/reproducible/debrepatch.git/tree/toolchain-patches

or as git commits here:

https://anonscm.debian.org/cgit/reproducible/gcc-6.git/
https://anonscm.debian.org/cgit/reproducible/dpkg.git/

I haven't yet filed bugs for them because they are not complete - e.g. they 
need documentation and upstream approval. But I'll do this soon in the next few 
weeks or so.

X

[1] 
https://tests.reproducible-builds.org/debian/issues/unstable/randomness_in_r_rdb_rds_databases_issue.html

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 101's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/101/

Feel free to commit fixes directly to drafts/101.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

gcc-6_6.3.0-11.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

package uploaded to our repo

[Ximin Luo]

dpkg_1.18.23.0~reproducible1.dsc has just been uploaded to 
https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 99's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/99/

Feel free to commit fixes directly to drafts/99.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 97's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/97/

Feel free to commit fixes directly to drafts/97.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

There were two duplicate "upcoming events" mentioned in both 97.mdwn and 
98.mdwn so I removed them for this current week, with the intention that they 
will be in there next week. I'm somewhat open to duplicating them though, if 
people feel strongly about it.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Please review the draft for week 95's blog post

[Ximin Luo]

Hi all,

This week's blog post draft is available for review:

https://reproducible.alioth.debian.org/blog/drafts/95/

Feel free to commit fixes directly to drafts/95.mdwn in

https://anonscm.debian.org/git/reproducible/blog.git/

I'll wait at least 24 hours from the time of this email for any comments, and 
if everything is good then I will publish it soon after that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: Bug#855282: debsign: support .buildinfo files

[Ximin Luo]

Control: tags + patch

Hi all,

I've done an initial implementation here:

https://anonscm.debian.org/cgit/collab-maint/devscripts.git/log/?h=pu/debsign-buildinfo

Please review!

I haven't yet updated debrsign but I think that program is a bit pointless 
anyway, and have documented this in debsign(1): "note that it is probably safer 
to have your trusted signing machine use \fBdebsign\fR to connect to the 
untrusted non-signing machine, rather than using \fBdebrsign\fR to make the 
connection in the reverse direction."

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#855282: debsign: support .buildinfo files

[Ximin Luo]

Package: devscripts
Version: 2.17.1
Severity: wishlist

Dear Maintainer,

dpkg since version 1.18.19 has been signing buildinfo files by default.
debsign at the moment will ignore these and leave them unsigned. It would be
good to support them.

Ximin

-- Package-specific info:

--- /etc/devscripts.conf ---

--- ~/.devscripts ---
Not present

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 
'testing-debug'), (300, 'unstable'), (200, 'experimental'), (1, 
'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.8.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages devscripts depends on:
ii  dpkg-dev 1.18.22
ii  libc62.24-9
ii  perl 5.24.1-1
pn  python3:any  

Versions of packages devscripts recommends:
ii  apt 1.4~rc1
ii  at  3.1.20-3
ii  curl7.52.1-2
ii  dctrl-tools 2.24-2
ii  debian-keyring  2017.01.20
ii  dput0.12.0
ii  equivs  2.0.9+nmu1
ii  fakeroot1.21-3.1
ii  file1:5.29-3
ii  gnupg   2.1.18-3
ii  gnupg2  2.1.18-3
ii  libdistro-info-perl 0.14
ii  libdpkg-perl1.18.22
ii  libencode-locale-perl   1.05-1
ii  libgit-wrapper-perl 0.047-1
ii  liblist-compare-perl0.53-1
ii  liblwp-protocol-https-perl  6.06-2
ii  libsoap-lite-perl   1.20-1
ii  liburi-perl 1.71-1
ii  libwww-perl 6.15-1
ii  licensecheck3.0.29-1
ii  lintian 2.5.50.1
ii  man-db  2.7.6.1-2
ii  patch   2.7.5-1
ii  patchutils  0.3.4-2
ii  python3-debian  0.1.30
ii  python3-magic   1:5.29-3
ii  sensible-utils  0.0.9
ii  strace  4.15-2
ii  unzip   6.0-21
ii  wdiff   1.2.2-2
ii  wget1.18-4
ii  xz-utils5.2.2-1.2

Versions of packages devscripts suggests:
ii  adequate 0.15.1
ii  autopkgtest  4.3
pn  bls-standalone   
ii  bsd-mailx [mailx]8.1.2-0.20160123cvs-3
ii  build-essential  12.3
pn  check-all-the-things 
pn  cvs-buildpackage 
pn  devscripts-el
ii  diffoscope   67
ii  disorderfs   0.5.1-1
pn  dose-extra   
pn  duck 
ii  faketime 0.9.6-7
ii  gnuplot  5.0.5+dfsg1-5
ii  gpgv 2.1.18-3
pn  how-can-i-help   
ii  libauthen-sasl-perl  2.1600-1
ii  libfile-desktopentry-perl0.22-1
pn  libnet-smtps-perl
pn  libterm-size-perl
ii  libtimedate-perl 2.3000-2
ii  libyaml-syck-perl1.29-1+b2
pn  mozilla-devscripts   
ii  mutt 1.7.2-1
ii  openssh-client [ssh-client]  1:7.4p1-6
ii  piuparts 0.75
pn  ratt 
ii  reprotest0.6
pn  svn-buildpackage 
pn  w3m  

-- no debconf information

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Bug#855273: diffoscope: still fails to clean up after SIGTERM

[Ximin Luo]

Mattia Rizzolo:
> Package: diffoscope
> Version: 77
> Severity: important
> 
> So, yesterday we tried to re-enable artifacts saving on jenkins, and the
> disc filled again because of GBs of temporary files left around.
> 
> In a log the only message I see is:
> 
> |Wed Feb 15 23:28:21 UTC 2017  I: diffoscope 77 will be used to compare the 
> two builds:
> |E: Caught signal ‘Terminated’
> |Thu Feb 16 03:30:35 UTC 2017  E: otb failed to build reproducibly in 
> experimental on i386.
> 
> I have yet to try to reproduce it this time (and weird, because in when
> I tried before reenabling the saving artifacts it did clean up for me).
> 
> [..]
> 

Where did you do this? On the page for otb on tests.r-b.org in the rbuild.log I 
see:

Mon Jan 23 04:11:07 UTC 2017  I: diffoscope 69 will be used to compare the two 
builds:
E: Caught signal ‘Terminated’: terminating immediately
E: Caught signal ‘Terminated’
Mon Jan 23 06:11:14 UTC 2017  E: otb failed to build reproducibly in unstable 
on amd64.

This possibly means diffoscope got a second SIGTERM whilst it was trying to 
clean up the first one. But this isn't present in your example above.

WARNING: You shouldn't trust packages downloaded from this host, they can 
contain malware or the worst of your fears, packaged nicely in debian format.
If you are aware of this and just want to use these artifacts to investigate 
why diffoscope 69 had issues, you can download the artifacts from the following 
location: 
https://tests.reproducible-builds.org/debian/artifacts/r00t-me/otb_unstable_amd64_tmp-LLmJ5/

I tried to download this URL but it looks like it's been deleted already. :(

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: diffoscope 77 in stretch or not?

[Ximin Luo]

Holger Levsen:
> On Tue, Feb 14, 2017 at 08:44:00PM +0000, Ximin Luo wrote:
>> I do think it's OK to try to support diffoscope 67 for 2 years because it's 
>> been quite well tested.
> 
> well, yes… but…
> 
>> I understand that 77 fixes quite a lot of bugs over 67…
> 
> 77 *exists* and is quite probably a lot better than 67, so I now think we
> should strive for 77 (or 77++ if needed) in stretch… I like 77, I just don't
> like the way we got there. But now that we have it, no need to hide it.
> 

I understand. Yeah, we don't need to change that. I was just voicing my opinion 
on "the way we got there" as well.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds