Re: Making schleuder build reproducibly
On Tue 2017-11-07 17:02:02 +0100, Georg Faerber wrote: > On 17-11-04 16:01:43, Holger Levsen wrote: >> On Mon, Oct 30, 2017 at 06:21:39PM +0100, Georg Faerber wrote: >> > @dkg: It seems, there is still a bug / race in dirmngr, which leads to >> > errors like "can't connect to '127.0.0.1': no IP address for host" and >> > in turn "marking host '127.0.0.1' as dead". See the attached debug log for >> > details, the log was taken on October 1st with dirmrngr out of unstable. >> > I'm happy to debug this further, if needed. >> >> indeed, random success+failure is visible for 3.2.1-1 on armhf: >> >> https://tests.reproducible-builds.org/debian/rb-pkg/buster/armhf/schleuder.html > > I'm actually unsure if these failures are caused by dirmngr, or if it's, > more likely, failing ruby code. I believe it's likely that your local keyserver isn't listening properly, but dirmngr's errors/warnings are certainly confusing. I've started a thread over on gnupg-devel to try to clean up the dirmngr error messages at least. If you can replicate that sequence of log messages (in particular, the "Unknown host" weirdness) with a minimal example on some system configuration, it'd be great to follow up over there. I've been unable to replicate it myself with dirmngr 2.2.2-1. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Question about strip-nondeterminism in bsh
On Thu 2017-10-12 03:02:49 +0100, Chris Lamb wrote: > The GitHub repositories are read-only mirrors. If you are checking out > the code as a member of the Reproducible Builds team, then it makes more > sense to get it from Alioth, for example: > > $ git clone git.debian.org:/git/reproducible/strip-nondeterminism or: debcheckout strip-nondeterminism That'll give you a local mirror that you can use to track upstream changes, and to make your own changes. But you won't be able to immediately push back to the upstream repo. If you've made commits and you want to push them upstream and you've got privileges to do so on alioth (i think you do, since "jathan-guest" is a member of "scm_reproducible"), you can enable the local repo for pushing like this: cd strip-nondeterminism git config remote.origin.pushurl git.debian.org:/git/reproducible/strip-nondeterminism hth, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#877418: dh-strip-nondeterminism: kills clojure performance
On Fri 2017-10-06 10:00:27 +0100, Chris Lamb wrote: > If it were hardcoded into the filenames, one wouldn't need to do > anything onerous, eg. > > -rw-r--r-- 1 0 Oct 6 09:56 > helloworld.adc83b19e793491b1c6ea0fd8b46cd9f32e592fc.class > -rw-r--r-- 1 0 Oct 6 09:56 > helloworld.adc83b19e793491b1c6ea0fd8b46cd9f32e592fc.clj > > (Not entirely serious) ah! i hadn't even thought of that :) I wonder whether any language would consider such a construct. > Just to underline, Python in Debian would not be a problem even with < > unless you consider building a .deb with SOURCE_DATE_EPOCH="$(date +%s)" > and installing that very same .deb within same second... > > … but I understand you were being more general about this topic! yep, exactly -- i'm not saying that python is broken in debian, just citing it as an example of another language that does the same kind of thing, similarly to elisp, etc. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#877418: dh-strip-nondeterminism: kills clojure performance
On Thu 2017-10-05 10:56:46 +0100, Chris Lamb wrote: > I'd also be curious to know why you think *more* than one second could > ever be needed here. I think I'm mising something. some filesystems have a resolution > 1s :( http://www.ntfs.com/exfat-comparison.htm shows that FAT32 has a 2s granularity when used without extensions. Looks like the Linux kernel remembers a 1sec granularity while still mounted, but shows just the 2sec granularity across remounts: mkfs -t vfat $blkdev mount $blkdev /mnt for a in 1 2 3; do touch /mnt/$a sleep 1 done stat /mnt/* | grep Modify umount /mnt mount $blkdev /mnt stat /mnt/* | grep Modify umount /mnt produces two batches of mtime stats: Modify: 2017-10-05 12:56:14.0 -0700 Modify: 2017-10-05 12:56:15.0 -0700 Modify: 2017-10-05 12:56:16.0 -0700 Modify: 2017-10-05 12:56:14.0 -0700 Modify: 2017-10-05 12:56:14.0 -0700 Modify: 2017-10-05 12:56:16.0 -0700 --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#877418: dh-strip-nondeterminism: kills clojure performance
On Wed 2017-10-04 19:45:49 +0100, Chris Lamb wrote: > *Very* quick thoughts here: could some variant of a) be merged > upstream…? Perhaps upstream could move to a hash-based system instead > of using timestamps? eg. encoding the SHA1 of the file in the filename. I'm thinking about this problem more generally than clojure specifically -- other folks have raised python's .py → .pyc mappings and i'm sure there are other similar frameworks. I want to make sure we're thinking about the various places that these checks happen. It may also matter whether we're talking about file stored in an archive vs. one stored in the filesystem. different archive formats and different filesystems have different timestamp granularity (iirc, FAT has 2s granularity, for example). And there are more questions too: what if multiple source files contributed to the creation of the compiled artifact (e.g. "include" directives)? You can also imagine a compilation regime that detects changes to a file (e.g. via inotify) and immediately triggers recompilation -- with a fast compiler and a coarse filesystem/archive timestamp, such a regime would end up in the same situation (serious performance impact). And of course, it's always possible to (accidentally or intentionally) just "touch" the timestamps on a totally different bytecode file of the appropriate name to trick or confuse this optimization step. There are also problems with the digest based approach that lamby suggests: it's significantly more expensive to do a full source extraction and digest than it is to compare timestamp metadata. -- So i think we have to ask what the goal of this check is from the upstream platform's point of view: * is it strong assurance that the file was built from the exposed source? * is it a speedy (if fallible) sanity check? i think that it can't really be the former (because of all the corner cases outlined above), so the question is what kind of failure modes and risks they're willing to tolerate. Those that want absolute assurance will be obliged to recompile each time unless they have some sort of externally-audited mapping/manifest. It sounds to me like python has made a sensible tradeoff (accepting that equal timestamps means OK) and clojure has made a decision that tries to get more of a guarantee than they can actually get, and sacrificed performance for it. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Bug#873937: dpkg: should include information about the used kernel in .buildinfo files
Over on 873...@bugs.debian.org, Holger Levsen wrote: > on reflection I agree that the privacy implications are too bad. > > On Fri, Sep 01, 2017 at 04:51:55PM +0200, Guillem Jover wrote: > [more insightful stuff I cannot disagree with removed.] > >> Given all the above, I'm inclined to just close the report? :) > > Probably, maybe, just please keep it open for another week or two for now, so > others can chime in… After reading Guillem's reasoning, i also agree that the report can be closed. Thanks for the thoughtful discussion, y'all. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] GCC build-path patch getting blocked
On Wed 2017-08-16 15:40:00 +, Ximin Luo wrote: > This idea is similar to hardening-wrapper. That is now deprecated, but > was useful as a stepping-stone to more "proper" fixes. Likewise, this > shouldn't be thought of as "the proper fix", but will give us some > useful data on how many packages are affected by which combinations of > (a), (b), (c) or (d). Depending on the number of packages we will have > to patch (with that 2-line patch), it will maybe give weight to future > attempts to have GCC support this env var - and then it would be easy, > since the core functionality would already be in there - or else > highlight the issue so that maintainers of those packages fix things > "properly". This sounds like a frustrating situation, Ximin, but i think your proposal is a good path forward. Thanks for sticking with it. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Reproducibility in Policy
Thanks for the proposal. I like it! A few nit-picks below: On Fri 2017-08-11 16:08:47 -0700, Sean Whitton wrote: > - a version of a source package unpacked at a given path; I don't like the idea of hard-coding a fixed build path requirement into debian policy. We're over 80% with variable build paths in unstable already, and i want to keep the pressure up on this. The build location should not influence the binary output. > repeatedly building the source package on the architecture with maybe s/on the architecture/on any machine of the same architecture/ ? all the best, --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#871244: diffoscope: support keybox files with kbxutil
On Mon 2017-08-07 13:33:00 +0200, Mattia Rizzolo wrote: > As you know we/I regularly backport diffoscope to Debian stable, so we > care about having those tools available there as well. > So, do you plan on making gnupg-utils available in stretch-backports > (with all the ongoing maintenance that such choice entails) or do you > think that having an alternate build-dependency+recommends is a way to > go here (possibly with a version restriction, e.g. > gnupg-utils | gnupg (<< 2.1.21-4~) > )? this looks dependency looks fine to me. I'd rather not commit to backporting gnupg to stretch-bpo just yet. > I'd like to avoid having a delta between buster and stretch-bpo, at > least not this early in the development cycle.. yep, that makes sense. > Also, do you plan on sending a patch for diffoscope? :D it's on my stack, but it won't be in the next week or two. if someone beats me to it i won't be upset :) --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#871244: diffoscope: support keybox files with kbxutil
Package: diffoscope Version: 84 Severity: wishlist keybox files are a file format specific to the GnuPG project. The gnupg-utils package (currently only in experimental, but hopefully soon to be moved to unstable) ships kbxutil, which should provide sufficient textual diffs to get a better hint at the changes. in debian stable, kbxutil used to be shipped in the gnupg package itself, but depending on gnupg itself will pull in the whole suite (including per-user daemons like gpg-agent and dirmngr, which aren't necessary for the public data manipulation needed by diffoscope), so gnupg-utils is probably a better way to go. --dkg -- System Information: Debian Release: buster/sid APT prefers testing-debug APT policy: (500, 'testing-debug'), (500, 'testing'), (500, 'oldstable'), (200, 'unstable-debug'), (200, 'unstable'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.11.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages diffoscope depends on: ii python33.5.3-3 ii python3-libarchive-c 2.1-3.1 ii python3-magic 1:5.30-1 ii python3-pkg-resources 36.0.1-1 Versions of packages diffoscope recommends: ii acl 2.2.52-3+b1 pn apktool pn binutils-multiarch ii bzip21.0.6-8.1 ii caca-utils 0.99.beta19-2+b2 pn colord ii default-jdk [java-sdk] 2:1.8-59 ii default-jdk-headless 2:1.8-59 pn device-tree-compiler pn docx2txt pn enjarify ii fontforge-extras 0.3-4 pn fp-utils ii genisoimage 9:1.1.11-3+b2 ii gettext 0.19.8.1-2+b1 ii ghc 8.0.1-17+b1 ii ghostscript 9.21~dfsg-1 pn giflib-tools ii gnupg2.1.21-4 ii imagemagick 8:6.9.7.4+dfsg-15 ii imagemagick-6.q16 [imagemagick] 8:6.9.7.4+dfsg-15 pn jsbeautifier pn libarchive-tools pn llvm ii mono-utils 4.6.2.7+dfsg-1 ii odt2txt 0.5-1+b2 ii oggvideotools0.9.1-4 ii openjdk-8-jdk [java-sdk] 8u141-b15-3 ii openssh-client 1:7.5p1-5 ii pdftk2.02-4+b2 ii pgpdump 0.32-0.1 ii poppler-utils0.48.0-2 pn python3-argcomplete ii python3-debian 0.1.30 pn python3-guestfs ii python3-progressbar 2.3-4 ii python3-rpm 4.12.0.2+dfsg1-2+b1 ii python3-tlsh 3.4.4+20151206-1+b3 ii r-base-core 3.3.3-1 ii rpm2cpio 4.12.0.2+dfsg1-2+b1 ii sng 1.1.0-1+b1 ii sqlite3 3.19.3-3 pn squashfs-tools ii tcpdump 4.9.0-3 ii unzip6.0-21 ii vim-common 2:8.0.0197-5 ii xxd 2:8.0.0197-5+b1 ii xz-utils 5.2.2-1.3 Versions of packages diffoscope suggests: ii libjs-jquery 3.1.1-2 -- no debconf information ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed 2017-06-21 15:42:07 +0100, Ian Jackson wrote: > This is a very useful concept but I suggest you give it a new name. > "binaries-attested upload" perhaps ? I like the idea that we should name this thing, but i'd call it something like a "source-only upload with .buildinfo" or "source+buildinfo upload" instead. > To me "source-only upload" means that there were no binaries built, > and therefore no information about binaries included in the upload. i tend to think "source-only" in this phrase applies to "upload", meaning that the upload doesn't include binaries, and what i'm uploading doesn't include binaries. i acknowledge that it also includes some stuff that isn't actually sources, but this is true of normal "source-only" uploads too -- for example, such uploads include cryptographic signatures and selected elements of the changelogs, which are also not sources. ☺, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed 2017-06-21 13:38:42 +0100, Ian Jackson wrote: > This is an interesting use case which dgit should support. cool! > But I think this is not what dgit push-source should do. Sean's > proposed dgit push-source does not do any kind of binary package > build. I think this is correct. But this means there are no binaries > and nothing for the .buildinfo to talk about. I don't know anything about "dgit push-source", so i defer to you on that. > Do the "source-only uploads" that you are talking about mention the > hashes of these locally-built .debs in their .buildinfo, then ? yes. when building foo version 1.2-3, the .changes file mentions only: - foo_1.2-3.dsc - foo_1.2.orig.tar.bz2 - foo_1.2-3.debian.tar.bz2 - foo_1.2-3_amd64.buildinfo and the buildinfo file mentions: - foo_1.2-3.dsc - libfoo_1.2-3.deb - foo-tools_1.2-3.deb I *do not* upload any of the *.deb files to the archive. > Certainly `dgit push' will not do anything to any .buildinfo you may > have. I think maybe that your use case should be supported by having > a version of dgit push which drops the .debs from the .changes, but > leaves the .buildinfo ? Is that how you construct these uploads now ? I really don't have to do anything manually. The standard dpkg-buildpackage toolchain does it for me if i pass --changes-option=-S -- it all works as i expect, and kudos to the dpkg developers for that :) > (Also: is there anything right now that verifies your assertions about > the .debs? Not that the lack of such a thing would make the > .buildinfos useless, but my experience is that without closing that > loop it is likely that the arrangements for generating the .buildinfo > are wrong somehow in a way we haven't spotted.) In a standard upload of the type i'm describing i've asserted: a) I built version 1.2-3 on amd64, and it should be included in debian b) here are the digests of the source code (including debian packaging) c) given this explicit set of build dependencies, here are the digests of the binary packages that were produced on my system. You say "verify my assertions about the .debs", i think you're talking about part (c), but there's nothing specifically to verify there. I'm saying to the world what *i* found when i built them. You want to tell me you found something different? fine! Now we have something to investigate. You found the same thing? Great, but that's a corroboration, not a verification. I agree with you that it'd be nice in the future to "close the loop" by having infrastructure that monitors all of these developer-generated .buildinfo files, compares them to the buildd-generated .buildinfo files, and provides some sort of interface for easy reasoning about them. and such infrastructure could well show that something is wrong with how we're generating .buildinfo files; that's fine, we'd then modify how we generate buildinfo files going forward to correct it, if necessary. Imagine a fancy console that a debian developer could pull up which shows a list of binary packages they submitted which differ from the one being shipped by the archive, and which build-dependencies it noticed were different (or, just shows a green light if it's the case all of their current uploads have been corroboratively rebuilt). cool, eh? Or some future, stricter debian variant might even want to only allow a package to enter the archive if the binary packages created by the buildd of the submitted architecture match the binaries claimed by the submitting developer (i'm *not* proposing this for debian today. it could introduce hassle and delay because of the concerns about build-dep synchronization that Adrian raises, and we don't have the workflow for it smooth enough yet). But i don't think that we need to officially "close the loop" in any fancy (or strict) way to warrant shipping .buildinfo files from developers. The fancy console i propose above (or anything like it) can only be built and used across the archive once we have shipped the .buildinfo files. Unnecessarily stripping .buildinfo files that we know about only delays that process. Regards, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed 2017-06-21 14:16:00 +0300, Adrian Bunk wrote: > On Wed, Jun 21, 2017 at 09:30:14AM +, Holger Levsen wrote: >> Hi, >> >> trigger warning: nitpicking. >> >> On Wed, Jun 21, 2017 at 11:34:17AM +0300, Adrian Bunk wrote: >> > > I do source-only uploads because i don't want the binaries built on my >> > > own personal infrastructure to reach the public. But i want to upload >> > > the .buildinfo because i want to provide a corroboration of what i >> > > *expect* the buildds to produce. >> > If you expect that, then your expectation is incorrect. >> >> I actually think that dkg's expectation is right, "just" that reality is >> wrong. >> >> The design of the Debian buildd network is from times when machines were much >> less powerful than what we have today and it shows. >> >> I'd rather have deterministic builds than the current unpredictable mess. > > I understand what you want, but using buildinfo is not a good idea here. > > Based on how many broken binaries get uploaded from developers, > the environment of the person uploading the sources is not a good > basis for determining what package versions to install when building > on the buildds. lest there be any misunderstanding: i am *not* suggesting that i want the build daemons to select their packages based on what's in my .buildinfo. Ximin's interpretation of my intent is the correct one: i want to see whether we manage to reproduce the same output. if the binary package outputs differ, and the installed build-deps differ, fine. that's data that someone tracking how things are built can use in a future analysis. if the binar package outputs do *not* differ, and the build-deps differ, that's also interesting information. my goal here isn't to use the build daemons as the r-b infrastructure -- we've already got the r-b infrastructure for that. :) But i'm happy to be able to see some corroborative (or anti-corroborative) .buildinfos published so that people who want to analyze them can do so. --dkg PS I fully agree that the right outcome for debian overall is to not allow binary uploads from anyone, unless they're granted special dispensation (e.g. toolchain porters), but that's getting far afield from this thread. signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Hi Ian-- On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: > A .buildinfo file is not useful for a source-only upload which is > veried to be identical to the intended source as present in the > uploader's version control (eg, by the use of dgit). > > Therefore, dgit should not include .buildinfos in source-only uploads > it performs. If dgit sees that a lower-layer tool like > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit > should strip it out of .changes. I often do source-only uploads which include the .buildinfo. I do source-only uploads because i don't want the binaries built on my own personal infrastructure to reach the public. But i want to upload the .buildinfo because i want to provide a corroboration of what i *expect* the buildds to produce. why wouldn't dgit take the same approach? stripping the .buildinfo from the .changes seems like a wasted shot at a potential corroboration. or am i misunderstanding the question here? --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Making schleuder build reproducibly
Hi Georg-- On Thu 2017-06-15 21:19:12 +0200, Georg Faerber wrote: > I really would like to make the build of schleuder, a gpg enabled > mailing list, reproducible. However, I'm a bit lost on my own, that's > why I'm searching for input with this mail: > > Some of the upstream provided tests check if importing a key from a > keyserver work. For this, a keyserver is started, bound to localhost to > mock the behaviour of the keyservers out there. This seems to work > sometimes [1], sometimes it fails, leading to > > [2] gpg: error retrieving 'ad...@example.org' via keyserver: No keyserver > available > > or > > [2] gpg: connecting dirmngr at > '/tmp/schleuder-test/example.org/list4/S.dirmngr' failed: IPC connect call > failed Hm, these seem like they could be failures of either: * dirmngr * the test/demo keyserver have you tried instrumenting either of these components of the test suite to log more data so that you can see why they're failing? for dimrngr, you'd do something like adding the following to dirmngr.conf in the example homedir: debug-level guru log-file /tmp/schleuder-test/example.org/list4/dirmngr.log debug 0x verbose and then in the event of a test failure, dump the contents of dirmngr.log I'm not sure how the test/demo keyserver is currently implemented so i'm not sure how to get better diagnostics from it. hth, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#858867: diffoscope: please support pcap files
On Tue 2017-03-28 03:39:18 -0500, Chris Lamb wrote: > tags 858867 + pending > thanks > > Implemented in Git: > > > https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=c5d01341055d0aa0552d08725a6b8e44255b0374 woo, thanks! in my testing before i saw this, i ended up using diff against the output of "tshark -V -r xx.pcap" (from the tshark package), but i think the tcpdump output you've got (with timestamps) is arguably more thorough. Knowing the right flags to pass for specific .pcap debugging might vary depending on what kind of network trace differences the user is interested in (e.g. in some cases, the network admin doesn't care about wall-clock time, they just care about delay gaps, so it'd be nice to have the initial wallclock in one place -- at the top of the dump -- and then just offsets following; or the network admin might not care about the payload, and just be interested in the IP or TCP or UDP flags), but i think that kind of nuance might not something that diffoscope can really account for, given its generic mandate. Anyway, if somoene in the future tests and decides that tshark is "better" for diffoscope's purposes than tcpdump, they now have a specific place to patch. thanks, Lamby! --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#858867: diffoscope: please support pcap files
Package: diffoscope Version: 78 Severity: wishlist i want to be able to do "diffoscope capture-1.pcap capture-2.cap" and get something meaningful. --dkg -- System Information: Debian Release: 9.0 APT prefers testing-debug APT policy: (500, 'testing-debug'), (500, 'testing'), (200, 'unstable-debug'), (200, 'unstable'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages diffoscope depends on: ii python3-libarchive-c 2.1-3.1 ii python3-magic 1:5.29-3 ii python3-pkg-resources 33.1.1-1 pn python3:any Versions of packages diffoscope recommends: ii acl 2.2.52-3+b1 pn apktool pn binutils-multiarch ii bzip21.0.6-8.1 ii caca-utils 0.99.beta19-2+b2 pn colord ii default-jdk [java-sdk] 2:1.8-58 ii default-jdk-headless 2:1.8-58 pn enjarify ii fontforge-extras 0.3-4 pn fp-utils ii genisoimage 9:1.1.11-3+b2 ii gettext 0.19.8.1-2 ii ghc 8.0.1-17+b1 ii ghostscript 9.20~dfsg-2 ii gnupg2.1.19-2 ii imagemagick 8:6.9.7.4+dfsg-2 ii imagemagick-6.q16 [imagemagick] 8:6.9.7.4+dfsg-2 pn jsbeautifier pn llvm ii mono-utils 4.6.2.7+dfsg-1 ii openjdk-8-jdk [java-sdk] 8u121-b13-4 ii openssh-client 1:7.4p1-9 ii pdftk2.02-4+b2 ii poppler-utils0.48.0-2 pn python3-argcomplete ii python3-debian 0.1.30 pn python3-guestfs ii python3-progressbar 2.3-4 ii python3-rpm 4.12.0.2+dfsg1-1 ii python3-tlsh 3.4.4+20151206-1+b2 ii rpm2cpio 4.12.0.2+dfsg1-1 ii sng 1.1.0-1+b1 ii sqlite3 3.16.2-3 pn squashfs-tools ii unzip6.0-21 ii vim-common 2:8.0.0197-3 ii xxd 2:8.0.0197-3 ii xz-utils 5.2.2-1.2+b1 Versions of packages diffoscope suggests: ii libjs-jquery 3.1.1-2 -- no debconf information ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Moving towards a deb-buildinfo(5) Format 1.0
On Sun 2017-02-19 16:34:45 -0500, Guillem Jover wrote: > On Sun, 2017-02-19 at 11:39:28 -0800, Vagrant Cascadian wrote: >> On 2017-02-19, Guillem Jover wrote: >> >> * .buildinfo files are not generated when creating source-only uploads >> > >> > Fixed. Now always generated. >> >> On a related note, is it currently possible to create a .buildinfo with >> both the source and binary, but a corresponding .changes with only the >> source? > > Yes, in the same way it's been possible to do so for source-only > uploads up-to-now when using just dpkg-buildpackage: > > $ dpkg-buildpackage --changes-option=-S Thanks for this note! this is very useful, and i wish i'd known about it long ago. It would have been part of my usual invocation already. Many thanks for pushing buildinfo this far into dpkg, i'm excited to see what we can do with it in the future. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Buildinfo in the Debian archive, updates
On Wed 2016-12-07 06:31:00 -0500, Ximin Luo wrote: > Separately regarding the ECC point, I don't think we can assume that > at this time because DDs still have non-ECC signatures, and are still > doing binary uploads with buildinfo files that we want to store. sure, but that's just one buildinfo file per package, compared to N buildinfo files for the build daemons on N different architectures. We can certainly give all the buildd's ECC keys if we're concerned about size in the archive. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Buildinfo in the Debian archive, updates
On Tue 2016-12-06 17:41:34 -0500, Jonathan McDowell wrote: > The storage of the hashes of the signed buildinfo files in Packages.gz > seems to be in order to deal with the fact that the signature is not > available elsewhere. If dkg's suggestion of using ECC signatures is > followed then some quick checking shows a signature size of 165 bytes > (when ASCII armoured). This seems sufficiently small to me that you > could just map it into a Signature: field at the end of the buildinfo > stanza within buildinfo.xz, with the bonus that at some point that would > allow for multiple such fields, all within the archive mirror network. I'd be wary about this "multiple such fields" bit. it seems likely that different buildinfo files will not match each other, even if the *output* is reproducible. This is because buildinfo files can capture some things that do not have an impact on the resultant binary artifacts. Otherwise, though, i agree with Jonathan that stuffing a small signature into the buildinfo file itself seems OK. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Buildinfo in the Debian archive, updates
Hi all-- On Mon 2016-11-14 12:13:00 -0500, Ximin Luo wrote: > A GPG signature with a 4096-bit key is about 800 bytes in base 64: > > http://ftp.debian.org/debian/dists/sid/ (has 2 signatures, if you use `gpg > --list-packets`) > http://ppa.launchpad.net/infinity0/rust-nightly/ubuntu/dists/yakkety/ > > so it would be about 600 bytes compressed. Across 24000 source packages, this > would be > > 600 * 24000 ~= 13.7MB > > per architecture (including for arch:all). This doesn't seem too bad actually. If size is a concern, we can make this much smaller by using ed25519 or ecdsa P256 signatures from the buildd's instead of RSA 4096. gpgv in Debian stretch is able to validate these signatures, and gpg in Debian stretch is able to produce them. This should reduce the total size of the signatures to about ~1MiB per architecture, if i'm calculating this correctly. > In total, this would be (9+13)*2 (one arch:all, one arch:$native) ~= > 50MB download, which considering the advantage of not having to > contact a 3rd party, I think is just about worth it, even if you only > want to rebuild a few packages. Can you explain this computation? i'm assuming 9 is the number of debian architectures. If an architectures builds both arch:all and arch:$native, it can do it in a single build, producing a single (signed) buildinfo file. right? so why the 2? so the extra cost of the signatures is ~9*13 -- one signed copy from each architecture's buildd, which is 104MB. Using smaller signing keys on the buildd's sounds better to me. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Moving towards a deb-buildinfo(5) Format 1.0
On Sun 2016-11-13 22:33:29 +0900, Chris Lamb wrote: >> Multiple builds of the same source package will set SOURCE_DATE_EPOCH to >> the same value but will result in a different Build-Date. > > … but that would mean that a reproducible build will result in .buildinfo > files with different contents (varying on Build-Date). > > That seems, at the very least, somewhat non-intuitive to me. It is definitely not what most of us initially expected, but it is actually what we want. i look at it this way: * Ideally, the generated binary packages are reproducible *even when the build environment changes*. For example, I build a package as the user "dkg" on machine "alice" in path /home/dkg/src/foo, and you build it as "lamby" on machine "bob" in path /home/lamby/work/foo/foo, and we should get the same outcome. * The buildinfo file documents things that *might* influence the build, but it also documents things that *should not* influence the build. Two differing buildinfo files that produced the same output effectively say "even when the build environment varies in the way that these two do, the package is still reproducible" * We actually don't want people to have to replicate the exact build environment to get a binary match. I think it was Ximin who pointed out: "all software is reproducible if you create an exact atom-by-atom copy of the original build computer before building". But that's not what we really mean by reproducible builds. In short, we *want* buildinfos to vary, while we want the generated binary artifacts to be reproducible. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] reproducible Debian: filter Environment section from buildinfo files
On Mon 2016-11-07 09:14:00 -0500, HW42 wrote: > 1) Currently the diff contains only stuff which should be fixed. If we >include .buildinfo differences it will include stuff which should not >be "fixed" (and can't be by the package maintainer). What if we only included the .buildinfo differences (clearly demarcated) if there was other stuff which should be fixed? And if nothing needs to be fixed, then don't show the buildinfo differences. That strikes me as potentially useful output for someone trying to diagnose a problem. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] submit signed .buildinfo files to buildinfo.debian.net
On Tue 2016-11-01 11:49:42 -0400, Daniel Shahaf wrote:
> Mattia Rizzolo wrote on Tue, Nov 01, 2016 at 11:18:37 +:
>> On Tue, Nov 01, 2016 at 11:12:44AM +, Chris Lamb wrote:
>> > Feel free to change to -f
>>
>> Holger changed that to ${HOSTNAME}, a variable that I've never
>> understood where it comes from
>
> It's defined by bash. (And not exported to the environment)
indeed:
0 dkg@alice:~$ posh -c 'echo $HOSTNAME'
0 dkg@alice:~$ dash -c 'echo $HOSTNAME'
0 dkg@alice:~$ bash -c 'echo $HOSTNAME'
alice
0 dkg@alice:~$
isn't hostname(1) a more reliable construct?
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Tue 2016-11-01 11:56:00 -0400, Ximin Luo wrote: > But the TIMESTAMP patch is for __TIMESTAMP__ whereas our > currently-existing Debian patches are only for __DATE__ and __TIME__. ah, right. sorry i totally missed that, and of course it makes sense. Perhaps you could call that out explicitly in SRD_Debug.txt, and modify the documentation related to SOURCE_DATE_EPOCH as well. > I was going to send the __FILE__ patch in a separate email later > yes. Just so it's a bit less overwhelming for the reviewers. If you're sending in the __TIMESTAMP__ changes and the =-selection changes, i don't think the __FILE__ changes are overwhelming. I'd suggest sending the entire series, perhaps as an e-mail thread (e.g. via git-send-email), with SRD_Debug.txt as an introduction e-mail/cover letter. That way the recipients see the individual changes in sensible, non-overwhelming pieces, but they also don't have to wait for receipt of the completed series in case they have time to review it in full. Thanks for this work -- it is a significant contribution to GCC, and to the momentum behind reproducible-builds in general. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
Hi Ximin-- On Tue 2016-11-01 09:52:00 -0400, Ximin Luo wrote: > Updated patches here: > > https://gist.github.com/infinity0/72aba22411a2e8baaae4649329f96643 > > Everyone please feel free to comment / review before I soon send them off to > GCC. This looks great, thank you! I have not tried running it myself yet, but the logic and the structure of the changes seem sensible. A few nit-picky comments below: --- Looking at what we're doing in debian, i see: https://sources.debian.net/src/gcc-6/6.2.0-9/debian/patches/gcc-SOURCE_DATE_EPOCH.diff/ https://sources.debian.net/src/gcc-6/6.2.0-9/debian/patches/gcc-SOURCE_DATE_EPOCH-doc.diff/ https://sources.debian.net/src/gcc-6/6.2.0-9/debian/patches/gcc-SOURCE_DATE_EPOCH-2.diff/ https://sources.debian.net/src/gcc-6/6.2.0-9/debian/patches/gcc-SOURCE_DATE_EPOCH-2-doc.diff/ Is there a reason that these changes differ from SOURCE_DATE_EPOCH_-_for_TIMESTAMP_macro.patch ? Or is it just divergence between gcc 6 and the master dev branch? --- Also, i note that the -doc.diff patches above are really helpful (though i see no reason to break them out separately from the code changes unless there's an upstream convention to do so; i think coupling documentation changes with code changes is a Good Thing). Maybe this SOURCE_PREFIX_MAP changeset can also include changes to gcc documentation? --- Your accompanying SRD_debug.txt describes the reasoning behind these changes well, thanks! But it says: I will also be submitting a follow-up patch to perform this behaviour for the __FILE__ macro as well, Unless i'm misunderstanding, you have already included a patch for the __FILE__ macro as well, so unless you plan on submitting these with some sort of deliberately delayed timing, i'd change the text in SRD_debug.txt to reflect the fact that this is a complete changeset that we expect to Do the Right Thing. --- Finally, SOURCE_PREFIX_MAP_-_rsplit.patch changes from splitting on the first "=" to splitting on the last "=". I think you've justified this change well in previous e-mails on this thread, but the justification doesn't appear in this patchset. If the gcc folks are going to see this series fresh, it'd be good to explain the change and justify it, as well as marking it clearly as optional. That is, if this change makes upstream balk at accepting the patchset, we want them to drop it and accept the rest, right? Thanks for your work on this, Ximin! --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] submit signed .buildinfo files to buildinfo.debian.net
On Mon 2016-10-31 20:46:14 -0400, Holger Levsen wrote: > On Mon, Oct 31, 2016 at 08:42:28PM -0400, Daniel Kahn Gillmor wrote: >> This is not a glitch at all, these are instructions that will make it >> work well with gpg 2.1.x, and are harmless in 1.4.x. Please keep it >> intact, since reasonable people will want to use modern GnuPG. :) > > reasonable people running services are waiting for gpg 2.1.x appearing on > stable(-backports) ;-) all in due time, holger, all in due time. 2.1.15-8 isn't even in testing yet. But i appreciate the vote of confidence ;) --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] submit signed .buildinfo files to buildinfo.debian.net
On Mon 2016-10-31 19:52:13 -0400, Holger Levsen wrote: > still, there is this glitch: > > gpg: skipping control `%no-ask-passphrase' () > gpg: skipping control `%no-protection' () > > Is that harmless? This is not a glitch at all, these are instructions that will make it work well with gpg 2.1.x, and are harmless in 1.4.x. Please keep it intact, since reasonable people will want to use modern GnuPG. :) --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] submit signed .buildinfo files to buildinfo.debian.net
On Mon 2016-10-31 19:00:18 -0400, Chris Lamb wrote: > Daniel Kahn Gillmor wrote: > >> > we might use gpg signing for other purposes, so I removed that >> > constraint… fwiw, i didn't say this ↑↑ but it looks like you're attributing it to me :/ >> "hostname -a", which was silently changed from "hostname -f" in the >> prior version. > > hostname -f returns just my local hostname whilst -a returns the > fully-qualified version with a domain name. I would far prefer seeing > the latter. I suspect you've got something funny going on in /etc/hosts or /etc/hostname. but the man page for hostname(1) says: -a, --alias Display the alias name of the host (if used). This option is deprecated and should not be used anymore. […] -f, --fqdn, --long Display the FQDN (Fully Qualified Domain Name). A FQDN consists of a short host name and the DNS domain name. Unless you are using bind or NIS for host lookups you can change the FQDN and the DNS domain name (which is part of the FQDN) in the /etc/hosts file. See the warnings in section THE FQDN above und use hostname --all-fqdns instead wherever possible. […] -A, --all-fqdns Displays all FQDNs of the machine. This option enumerates all configured network addresses on all configured network inter‐ faces, and translates them to DNS domain names. Addresses that cannot be translated (i.e. because they do not have an appropri‐ ate reverse IP entry) are skipped. Note that different addresses may resolve to the same name, therefore the output may contain duplicate entries. Do not make any assumptions about the order of the output. i think we want -f, fwiw. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] submit signed .buildinfo files to buildinfo.debian.net
On Mon 2016-10-31 17:43:16 -0400, Holger Levsen wrote: > On Sat, Oct 29, 2016 at 11:28:46AM +0100, Chris Lamb wrote: >> Updated sign-buildinfo-submissions-with-gpg-key. I didn't squash & >> force push so that dkg's contribution is correctly attributed. :) > > cool! > >> (I *think* I'm understanding how & where update_jdn.sh is called, so >> please check the conditional makes sense; we want the individual build >> notes to sign the keys, not the jenkins master). > > thanks a lot! it just needed a small > > we might use gpg signing for other purposes, so I removed that > constraint… > > this is the result, seems to work nicely except one detail: > > gpg: directory `/var/lib/jenkins/.gnupg' created > gpg: new configuration file `/var/lib/jenkins/.gnupg/gpg.conf' created > gpg: WARNING: options in `/var/lib/jenkins/.gnupg/gpg.conf' are not yet > active during this run > gpg: keyring `/var/lib/jenkins/.gnupg/secring.gpg' created > gpg: keyring `/var/lib/jenkins/.gnupg/pubring.gpg' created > gpg: /var/lib/jenkins/.gnupg/trustdb.gpg: trustdb created > wbq0: Mon Oct 31 21:38:09 UTC 2016 - Generating GPG key for jenkins > user. > gpg: skipping control `%no-ask-passphrase' () > gpg: skipping control `%no-protection' () > ...+ I see no errors here. was there a hang or something? If so, maybe that machine is low on entropy? > gpg: directory `/var/lib/jenkins/.gnupg' created > gpg: new configuration file `/var/lib/jenkins/.gnupg/gpg.conf' created > gpg: WARNING: options in `/var/lib/jenkins/.gnupg/gpg.conf' are not yet > active during this run > gpg: keyring `/var/lib/jenkins/.gnupg/secring.gpg' created > gpg: keyring `/var/lib/jenkins/.gnupg/pubring.gpg' created > gpg: /var/lib/jenkins/.gnupg/trustdb.gpg: trustdb created > jtk1a: Mon Oct 31 21:37:54 UTC 2016 - Generating GPG key for jenkins > user. > gpg: -:4: missing argument I'm totally confused by this one. these are the same commands running on each host, right? line for of stdin in this case was (if i'm understanding fadd6ac719be7514ee0f54a787b36df81fda17df correctly) "hostname -a", which was silently changed from "hostname -f" in the prior version. i dunno why anyone cares about -a (hostname(1) suggests that it is deprecated), but if you want to use it, and it returns the empty string on some hosts, you can prefix the whole thing with something like "Name-Real: r-b builder $(hostname -a)" so that it's guaranteed to not be the empty string. really happy to see this moving along! thanks to lamby and holger for pushing on it. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [PATCH] Make use of gpg more flexible
On Fri 2016-10-28 17:17:10 -0400, Holger Levsen wrote: > thanks for chiming in…! > > On Fri, Oct 28, 2016 at 01:51:30PM -0400, Daniel Kahn Gillmor wrote: >> This set of commands should work with modern versions of gpg (2.1.x) >> as well, and should be independent of potentially variable output. > > does it also work with jessie? jenkins.d.(n|o) runs jessie, plus > jessie-backports… It should work with jessie -- i tried to only make changes that were backward-compatible. If that's not the case, please let me know. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[PATCH] Make use of gpg more flexible
This set of commands should work with modern versions of gpg (2.1.x)
as well, and should be independent of potentially variable output.
Additionally, we want the key to be signing-capable, but nothing else.
We also have no need to generate an encryption-capable subkey, so just
drop that part.
---
bin/reproducible_build.sh | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/bin/reproducible_build.sh b/bin/reproducible_build.sh
index dda7bba..d4d2e58 100755
--- a/bin/reproducible_build.sh
+++ b/bin/reproducible_build.sh
@@ -775,19 +775,19 @@ check_buildinfo() {
sign_buildinfo() {
# Greate GPG key if it does not already exist
- if ! gpg --list-secret-keys | grep -qs '^sec' >/dev/null 2>&1
+ if ! gpg --with-colons --fixed-list-mode --list-secret-keys | cut -d:
-f1 | grep -qsFx 'sec' >/dev/null 2>&1
then
log_info "Generating GPG key"
- gpg --batch --gen-key
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Thu 2016-10-27 11:18:00 -0400, Ximin Luo wrote: > So, I can implement option B fairly easily, but there is one ugly fact > about the syntax that we should think about. The way that GCC does it, > it means you can't strip a prefix that itself contains the "=" > character. > > Now this is not going to matter in all the cases that *we* want to use > it for. Doing a `sudo find / -name "*=*"` on my system gets me one > result: "songs/releases/2012 - Grimes - Visions/06 Vowels = space and > time.mp3". > > However, we're defining a *standard* here and if it takes off, it will > last for quite some time and have quite wide reach. It means that if > someone wants to have a folder structure like e.g. > > C:\Documents and Settings\Betrand Russell\Proofs of 1+1 = 2\Automated > Proofs\Source Code\ > > or perhaps in the future: > > /srv/net/distributed-hash-table/address/VaIWP8YIWDChR2O76yiufXBsbw5g2skB/kp9VP+qVLvydovdGw==/projects/gcc-6/ note that your "convenient" use of base64 here is problematic for other reasons, because of the "/" character -- some "uniform" paths will be more layers deep than others. This isn't directly relevant to our specific problem, but just an observation that stuffing b64 encodings into the filesystem might already be problematic. I guess quoted-printable encodings are even more likely to have problems with "=" though: 0 dkg@alice:~$ qprint -e <<<"abcdé" abcd=C3=A9 0 dkg@alice:~$ > these people won't be able to use this standard. Do we care about > these sorts of things? Even if not, we should explicitly say "Paths > containing '=' are not supported" - there is no sense in trying to > brush this fact "underneath the carpet". I agree that it is actively good to explicitly state this constraint. The constraint already exists (for debug-prefix-map). And "=" in filenames have other weird consequences that people might want to avoid, like this: mkdir foo=bar ln -s /bin/ls foo=bar/ foo=bar/ls -la the shell treats this as variable assignment and returns an error: bash: -la: command not found while this works fine: ./foo=bar/ls -la Anyway, i'm OK with saying "Building in a path that contains an '=' character is explicitly not supported" >> One other detail: Should we specify something about trailing '/' in >> oldprefix. This can be relevant when mapping directories. > > Did you have a specific example in mind? I think the current behaviour > is fine, especially if we support the arbitrary newprefix: if you give > it "/build/pkg/=." then you'll end up with paths like ".src/blah.c", > but if you give it "/build/pkg/=" then the result is fine. fwiw, i think the fact that these are raw strings is fine. the same rules apply for "git format --prefix" and that seems to work out OK. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Tue 2016-10-25 08:29:00 -0400, Ximin Luo wrote:
> Option A
>
> oldprefix = getenv("SOURCE_ROOT_DIR")
> newprefix = getenv("SOURCE_ROOT_PREFIX") or "."
>
> Pros: Simple, easy to understand. Works almost exactly as debug-prefix-map
> which has already existed for ages in GCC.
> Cons: Uses two environment variables.
>
> Option B
>
> oldprefix, newprefix = getenv("SOURCE_PREFIX_MAP").split("=", 1)
>
> Pros: Works *exactly* as debug-prefix-map
> Cons: Have to parse the thing. GCC splits on the first "=" but I found this
> out by looking at the source code (gcc/final.c:add_debug_prefix_map), it's
> not explicitly documented. Someone else could split at the last "=" for
> example.
While i think A would be simpler in a perfect world with no history, i
think i prefer B here.
We'll be documenting this, so we can simply define which "=" character
we split on. I'd even be fine with saying that we don't allow "=" in
either path, fwiw, if it makes things easier.
B has the advantages of being able to justify it with reference to
known, existing functionality (debug-prefix-map) and the environment
variable name is clearer (none of the "ROOT" ambiguity).
> Option C
>
> oldprefix = getenv("ALLSOURCES_DIR")
> newprefix = "."
>
> Oh em gee what is going on here? Well this is a slight hack. Instead of
> setting SOURCE_ROOT_DIR=/build/blahblah we set ALLSOURCES_DIR=/build so that
> "blahblah" remains in the build output. This is analogous to the difference
> between "datadir = /usr/share/@PACKAGE@" vs "datarootdir = /usr/share" (in
> GNU installation directories naming conventions).
>
> Pros: Simple for implementers, one environment variable that doesn't have to
> be parsed.
> Cons: Requires reproducers to do the build in a directory called "blahblah".
> At least, everyone can do this - if you can't create directories with
> arbitrary names, you probably can't set environment variables either. However
> it does mix up two different mechanisms (env var + directory naming) to
> achieve reproducibility.
This also makes it harder to do a reproducible build from a VCS
checkout, fwiw, if the build path is going to have the version number in
it. the VCS checkout is likely to just be in $PACKAGENAME, and having
to rename it to $PACKAGENAME-$VERSION just to do the build seems weird
and potentially problematic. We want it to be easy to reproduce, with
minimal hoop-jumping.
Thanks for laying the options out clearly, Ximin.
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Mon 2016-10-24 16:18:00 -0400, Ximin Luo wrote: > Almost, with two minor corrections: > > I picked SOURCE_ROOT_DIR; SOURCE_ROOT has 6395 pages of results in > sources.debian.net and I didn't want to check that none of these are > environment variables. fair enough. > SOURCE_ROOT_PREFIX would only be prepended if SOURCE_ROOT_DIR is set > and was stripped. It's not prepended to all paths; and setting it by > itself without SOURCE_ROOT_DIR wouldn't do anything (since nothing > gets stripped in that case). What if SOURCE_ROOT_DIR is not an actual prefix of the current file. like if SOURCE_ROOT_DIR=/no/such/path and the package is built in /home/user/src/foo ? > I would also say "absolute path" instead of "source-related path" > since the latter could be misinterpreted to exclude intermediate build > files. I used the term "source-related" to try to refer to the build tree. I wouldn't want to give the impression that *any* absolute path that makes it into the output artifacts will be stripped or prefixed. For example, /usr/bin/gpg embeds absolute paths like /usr/share/gnupg and /dev/tty, but these are not "source-related". We need to find a clear, concise way to explain the proposal that won't confuse people. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Mon 2016-10-24 15:33:00 -0400, Ximin Luo wrote:
> HW42:
>> Btw: Do you know why currently debug-prefix-map maps the source dir to
>> '.'? (My guess is because that's the easiest in dpkg-buildflags) I think
>> for debugging between package boundaries '${srcpkg}-${srcver}' would be
>> much better.
>>
>> So maybe a MAP_SOURCE_PATHS variable would be better. But this has the
>> big disadvantage of making things more complex.
>
> I think this idea is good in principle and we should try to aim for
> it. My preference would be to have a second environment variable
> though, e.g. SOURCE_ROOT_PREFIX.
>
> I think combining both of these envvars into one single envvar of the
> form "$source=$target", though it matches the syntax for
> -fdebug-prefix-map, would make it harder for other programs and build
> tools to adopt as a standard.
Let me try to restate this cleanly. The goal is to standardize the
pathnames for any source-related path that gets embedded in the output
by compilation or other build-time processing.
SOURCE_ROOT → (optional) The full path to the base of the unpacked
source tree. This will be stripped from the beginning of
any source-related path that makes its way into any
generated artifact.
SOURCE_ROOT_PREFIX → (optional) This string is prefixed to the
beginning of any source-related path that makes
its way into any generated artifact.
SOURCE_ROOT path mangling takes place before SOURCE_ROOT_PREFIX path
mangling.
Is that the current proposal?
--dkg
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [rb-general] Scheduling a regular time for IRC meetings (#2)
On Tue 2016-10-18 07:47:00 -0400, Chris Lamb wrote: > The consensus was that we want to try polling again including more range of > times. I will close this poll in one week from today and then send out a mail > scheduling the first one. Just to clarify: these times represent times for a regular IRC meeting related to reproducible builds -- but are they expected to be weekly meetings? fortnightly meetings? monthly meetings? iirc, we were talking about roughly fortnightly (every two weeks) or semimonthly (twice a month). in other words, we're not expecting people to make themselves available at times they agree to here every single week. right? --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: opi2a back! init system variations?
On Wed 2016-10-05 23:42:17 -0400, Vagrant Cascadian wrote: > The whole ordeal did trigger a thought about adding init system > variations to the builds... at least could do systemd and sysvinit at > the moment without a *huge* amount of trouble, unless we're relying on > some init-specific features for the builders? fwiw, i imagine that the choice of what is pid 1 (much like the choice of what is /bin/sh) could be a significant factor in some builds. I'd be happy to see this variation added to the list of things we'd like to eventually test. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Fri 2016-09-09 01:33:00 +0200, HW42 wrote:
> According to codesearch.d.n it seems only SOURCE_DIR_ROOT isn't taken
> yet. (Google finds at least one case).
thanks for looking, that's what i should have done in the first place.
> also bikeshedding: I dislike SOURCE_DIR_ROOT because the name is some
> how redundant. What's the difference between SOURCE_DIR and
> SOURCE_DIR_ROOT? But I don't care much about the naming.
if i've got some source with code in three directories, they're all
SOURCE_DIRs, but only the top level is SOURCE_DIR_ROOT.
> Btw: Do you know why currently debug-prefix-map maps the source dir to
> '.'? (My guess is because that's the easiest in dpkg-buildflags) I think
> for debugging between package boundaries '${srcpkg}-${srcver}' would be
> much better.
I guess i don't have much of an opinion either way here, as long as it
only adds noise to the -dbgsym packages.
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Tue 2016-09-06 16:02:00 -0400, Ximin Luo wrote: > Thanks, I did see this a while ago and forgot about it. However it > does differ from the current proposal in an important way. > > Current proposal (2): GCC should, if SOURCE_ROOT is set and > debug-prefix-map is not given, *automatically* use this > variable. There is no opportunity for the user to tell GCC to look at > a different variable. > > OTOH your patch above has GCC read a user-supplied variable. I think > we want to avoid this, for the same reason that we pushed quite > heavily for upstreams to support SOURCE_DATE_EPOCH and not their own > custom command line option. It would also fix packages using gcc but > not dpkg-buildflags (and for other distros, etc). ah, right. I like your constrained version better; hard-coding sensible practices helps keep everyone on the right track. It's also much simpler to argue for this approach if we can point to how SOURCE_DATE_EPOCH is already supported. bikeshedding: is SOURCE_ROOT the right name? I worry a bit about the different possible meanings of "ROOT" on unix systems. maybe SOURCE_BASE_DIR or SOURCE_DIR_ROOT is more descriptive of what we are describing? No strong preferences on my side. > But for sure we can start from the code that you already wrote. :) yep, that's what i was proposing; i certainly don't mean to suggest that my original patch is the thing that should be adopted. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Minimising work needed for this build-path issue
On Wed 2016-08-24 07:20:00 -0400, Ximin Luo wrote: > 2. Define another variable SOURCE_ROOT to be set to the top-level > source dir, and patch GCC to use this as the default value for > debug-prefix-map (and the analogue for other languages / tools). > > This would have the same concrete behaviour as the current situation, > but then we're defining yet another variable... but probably less > tools will need to support this than SOURCE_DATE_EPOCH. And as with > (1), this would not be necessary for the path-is-namespace languages. Please see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68848 in particular, this patch here: https://gcc.gnu.org/bugzilla/attachment.cgi?id=37005 See also the mild pushback from upstream about the env var implementation: Bernd Schmidt wrote: >>> Wouldn't it be simpler just to special-case -fdebug-prefix-map in >>> gen_producer_string? The environment variable thing strikes me as >>> unnecessary. https://gcc.gnu.org/ml/gcc-patches/2015-12/msg01280.html I didn't have Ximin's analysis to back up the argument for the env var case, and it seemed (and was) easier to just drop -fdebug-prefix-map From gen_producer_string. If you want to try to push this forward, i can follow up on that thread and ask for reconsideration of the associated patch. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Bug#835465: [Reproducible-builds] Bug#835465: python-apt: FTBFS: AptKeyError: recv from 'hkp://localhost:19191' failed for '0xa1bD8E9D78F7FE5C3E65D8AF8B48AD6246925553'
On Tue 2016-08-30 10:31:50 -0400, Julian Andres Klode wrote: > On Tue, Aug 30, 2016 at 10:18:33AM -0400, Daniel Kahn Gillmor wrote: >> Can you point me to a report about that? I'm not seeing it in my scan >> of the bts at https://bugs.debian.org/src:software-properties > > There is none. And it does not seem to be that annoying. While it now only > shows key ids and no names, you can at least still remove them. So it > basically > works, but without it showing names, it's kind of useless for the target > audience. ok, i'm not going to worry about marking software-properties with a Breaks: unless i hear otherwise from you. thanks for giving me a heads-up, though. It does look like there's code in there that should probably be pulled out just have it use python-apt directly. > APT uses gpgv and then parses the output of [GNUPG:] VALIDSIG to > determine the key id used to sign (and forgets to include the > subkey field). > > If a list of allowed keys has been specified for the source via a > Signed-By field in the sources.list or the Release file, it then > checks that the key it parsed from GPG is in that list. > > If nobody specifies a Signed-By field (which is the default > right now), that of course does not affect anyone. I do want > to roll out the Release-file Signed-By field at some point > though, as it (very) slightly improves security. Huh, i was unaware of the Signed-By field, actually, but i'm happy to see it. I'm not sure what i think about a primary key being listed there by fingerprint while the Release file being signed by a subkey. I could see a (fairly weak) argument that rejecting that signature is "working as it should", but i think you're probably right that this might be better considered as a bug. at least it's a bug that fails closed rather than failing open though :) --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Bug#835465: [Reproducible-builds] Bug#835465: python-apt: FTBFS: AptKeyError: recv from 'hkp://localhost:19191' failed for '0xa1bD8E9D78F7FE5C3E65D8AF8B48AD6246925553'
On Tue 2016-08-30 09:47:34 -0400, Julian Andres Klode wrote: > On Tue, Aug 30, 2016 at 09:32:15AM -0400, Daniel Kahn Gillmor wrote: >> On Tue 2016-08-30 08:49:20 -0400, Julian Andres Klode wrote: >> >> apt/auth.py appears to want to force gnupg to store its secret key >> >> material in secring.gpg. This isn't a best practice, and modern >> >> versions of gpg do not do so by default. I'd recommend dropping >> >> tmp_secret_keyring entirely. >> > >> > Hmm, there should not even be any secret key material, as apt only >> > deals with public keys. >> >> agreed, all the more reason to strip out those extra directives ;) > > I assume that might change behavior if used with gpg1, so I'd rather > keep it in if it does no harm. There should be no behavior that changes based on secret keys if secret keys are never used. I'd rather not have spare directives floating around that we don't need -- i've been doing triage on things that try to manipulate gnupg programmatically, and the simpler we can make them the easier it will be to fix any problems that come up in GnuPG itself, i think. There's a lot of cruft, and it would help my sanity to minimize the cruft. > I don't really care. More important are probably Breaks for > software-properties, > it's Authentication tab is fairly broken now. I think that's also where > apt.auth > was split off from, but I'm not entirely sure... Can you point me to a report about that? I'm not seeing it in my scan of the bts at https://bugs.debian.org/src:software-properties >> > Maybe there's also an option to display fingerprints instead of keyids >> > in --with-colons --list-keys? >> >> sure! >> >> gpg --fixed-list-mode --with-fingerprint --with-fingerprint --with-colons >> --list-keys >> >> will produce lines of the form: >> >> fpr:0EE5BE979282D80B9F7540F1CCD2ED94D21739E9: > > How awful. There should be a way to integrate this into the pub output To be clear, the output has pub:, then fpr:, then uid: lines in a row. it's pretty straightforward to track as you read the lines linearly. for any uid line, it is associated with the most recent fpr line, which itself is associated with the most recent pub line. the uid line is split out because you can have multiple uids associated with each pub+fpr. For the fields we're interested in, this is the same output across all versions. > (and with all the other breaks, it should have just gone fingerprint > by default everywhere). i'm working on that, but there are internal data structure considerations that make it more costly to display the fingerprint (unfortunately, the datastructures in the keyring are 64-bit keyids, not full fingerprints). > APT's subkey handling is fairly broken anyway (it's gpg verification > does not consider subkeys at all, so if you specify a list signed-by > of master key ids, APT would fail to validate a repo signed with a > subkey, unless the subkey is in the list itself...) interesting -- how is this done? i thought apt was using gpgv to verify the signatures, and if there are subkeys in the keyrings gpgv knows about gpgv will be willing to accept those subkeys. Are you saying apt itself parses the output of gpgv and the fingprints in it to some internal list of acceptable fingerprints? If you could point to the right spot in the source, i'd be happy to look into it further. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Bug#835465: [Reproducible-builds] Bug#835465: python-apt: FTBFS: AptKeyError: recv from 'hkp://localhost:19191' failed for '0xa1bD8E9D78F7FE5C3E65D8AF8B48AD6246925553'
On Tue 2016-08-30 08:49:20 -0400, Julian Andres Klode wrote:
>> apt/auth.py appears to want to force gnupg to store its secret key
>> material in secring.gpg. This isn't a best practice, and modern
>> versions of gpg do not do so by default. I'd recommend dropping
>> tmp_secret_keyring entirely.
>
> Hmm, there should not even be any secret key material, as apt only
> deals with public keys.
agreed, all the more reason to strip out those extra directives ;)
>> I'll be releasing a new version of gnupg shortly that will explicitly
>> declare that it Breaks: python-apt (<= 1.1.0~beta4).
>
> I think that's a bit overkill. While this part of python-apt is broken
> with the new gnupg, the rest works fine; and nobody uses the apt.auth
> module. Not to mention that I'm deprecating it, as we deprecated the gpg
> stuff in apt-key.
If you want me to remove the Breaks: i can do so -- my goal was to
address the concerns raised in https://bugs.debian.org/835349.
If you'd rather that i not provide a Breaks: or a Conflicts: for
python-apt, i can avoid it -- speak up though, i'm hoping to release the
next version of gnupg2 to unstable shortly :)
>> Ideally, the next version of python-apt can have these bugs fixed and it
>> will work cleanly with the modern version of gnupg.
>
> Sure. But we should really support both old and new gpg versions, otherwise
> it gets a bit annoying.
>
> Maybe there's also an option to display fingerprints instead of keyids
> in --with-colons --list-keys?
sure!
gpg --fixed-list-mode --with-fingerprint --with-fingerprint --with-colons
--list-keys
will produce lines of the form:
fpr:0EE5BE979282D80B9F7540F1CCD2ED94D21739E9:
The hex string shows up in $10 for "awk -F:", fields[9] in python after
fields = line.split(":").
providingn --with-fingerprint twice ensures that you get fingerprints
for both primary keys and subkeys -- if that's what you want.
>> However, if your next upload of python-apt can't be built or run against
>> modern versions of GnuPG
>
> That would be silly :)
i'm glad it will be straightforward to sort it out ;)
Thanks for your work on this,
--dkg
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#835075: libmail-gnupg-perl: FTBFS: Failed 1/10 test programs. 0/4 subtests failed.
Control: affects 835075 src:gnupg2 On Mon 2016-08-22 03:20:05 -0400, Chris Lamb wrote: > Source: libmail-gnupg-perl > Version: 0.22-1 > Severity: serious > Justification: fails to build from source > User: reproducible-builds@lists.alioth.debian.org > Usertags: ftbfs > X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org > > Dear Maintainer, > > libmail-gnupg-perl fails to build from source in unstable/amd64: > > [..] > > > > ** > ** Starting build > ** > > ** > >Package: libmail-gnupg-perl >Version: 0.22-1 >Build architecture: amd64 >Date: Mon, 22 Aug 2016 08:12:30 +0100 >Hostname: 0ca47040 >Uname:Linux 0ca47040 4.6.0-1-amd64 #1 SMP Debian > 4.6.4-1 (2016-07-18) x86_64 GNU/Linux >/etc/timezone:Europe/London > > > ** > ** Installing build dependencies > ** > > ** > > dh_testdir > dh_testroot > dh_prep > dh_testdir > dh_testroot > dh_install > dh_installdocs > dh_installchangelogs > dh_compress > dh_fixperms > dh_installdeb > dh_gencontrol > dh_md5sums > dh_builddeb > dpkg-deb: building package 'libmail-gnupg-perl-build-deps' in > '../libmail-gnupg-perl-build-deps_0.22-1_all.deb'. > > The package has been created. > Attention, the package has been created in the current directory, > not in ".." as indicated by the message above! > Selecting previously unselected package libmail-gnupg-perl-build-deps. > (Reading database ... 23244 files and directories currently installed.) > Preparing to unpack libmail-gnupg-perl-build-deps_0.22-1_all.deb ... > Unpacking libmail-gnupg-perl-build-deps (0.22-1) ... > Reading package lists... > Building dependency tree... > Reading state information... > Correcting dependencies... Done > The following additional packages will be installed: > libclass-method-modifiers-perl libconvert-binhex-perl libdata-perl-perl > libdevel-globaldestruction-perl libgnupg-interface-perl > libimport-into-perl > libmime-tools-perl libmodule-build-perl libmodule-runtime-perl libmoo-perl > libmoox-handlesvia-perl libmoox-late-perl libparams-classify-perl > librole-tiny-perl libstrictures-perl libsub-exporter-progressive-perl > libtest-pod-perl libtype-tiny-perl > Suggested packages: > libscalar-number-perl libbareword-filehandles-perl libindirect-perl > libmultidimensional-perl libdevel-stacktrace-perl > Recommended packages: > libmodule-signature-perl libpod-readme-perl libsoftware-license-perl > libclass-xsaccessor-perl libdevel-lexalias-perl libtype-tiny-xs-perl > The following NEW packages will be installed: > libclass-method-modifiers-perl libconvert-binhex-perl libdata-perl-perl > libdevel-globaldestruction-perl libgnupg-interface-perl > libimport-into-perl > libmime-tools-perl libmodule-build-perl libmodule-runtime-perl libmoo-perl > libmoox-handlesvia-perl libmoox-late-perl libparams-classify-perl > librole-tiny-perl libstrictures-perl libsub-exporter-progressive-perl > libtest-pod-perl libtype-tiny-perl > 0 upgraded, 18 newly installed, 0 to remove and 0 not upgraded. > 1 not fully installed or removed. > Need to get 1148 kB of archives. > After this operation, 2785 kB of additional disk space will be used. > Get:1 http://httpredir.debian.org/debian sid/main amd64 > libmodule-build-perl all 0.421800-2 [259 kB] > Get:2 http://httpredir.debian.org/debian sid/main amd64 > libclass-method-modifiers-perl all 2.12-1 [18.6 kB] > Get:3 http://httpredir.debian.org/debian sid/main amd64 > libsub-exporter-progressive-perl all 0.001011-1 [7748 B] > Get:4 http://httpredir.debian.org/debian sid/main amd64 > libdevel-globaldestruction-perl all 0.13-1 [8676 B] > Get:5 http://httpredir.debian.org/debian sid/main amd64 > libparams-classify-perl amd64 0.013-6 [24.2 kB] > Get:6 http://httpredir.debian.org/debian sid/main amd64 > libmodule-runtime-perl all 0.014-2 [18.0 kB] > Get:7 http://httpredir.debian.org/debian sid/main amd64 libimport-into-perl > all 1.002005-1 [11.6 kB] > Get:8 http://httpredir.debian.org/debian sid/main amd64 librole-tiny-perl > all 2.03-1 [19.0 kB] > Get:9 http://httpredir.debian.org/debian sid/main amd64 libstrictures-perl > all 2.03-1 [18.1 kB] > Get:10 http://httpredir.debian.org/debian sid/main amd64 libmoo-perl all > 2.002004-1 [66.8 kB] > Get:11 http://httpredir.debian.org/debian sid/main amd64 libdata-perl-
Re: Bug#835592: php-crypt-gpg: FTBFS: Tests: 286, Assertions: 514, Errors: 77, Failures: 8, Skipped: 8
On Tue 2016-08-30 06:49:30 -0400, Axel Beckert wrote: > Daniel Kahn Gillmor wrote: >> However, if your next upload of php-crypt-gpg can't be built or run >> against modern versions of GnuPG, then you probably need to state this >> package's dependency on gnupg as gnupg (<= 2.1). > > Shouldn't "gnupg1" be used then nowadays? that would involve a fix of forcing php-crypt-gpg to use and look for /usr/bin/gpg1. that would make me sad, because it would mean that the users of php-crypt-gpg would be unable to benefit from the work happening on GnuPG's modern branch. and the fixes available in gcrypt itself. So yeah, it's possible to do it that way but i really hope it doesn't come to that. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#835465: python-apt: FTBFS: AptKeyError: recv from 'hkp://localhost:19191' failed for '0xa1bD8E9D78F7FE5C3E65D8AF8B48AD6246925553'
Control: affects 835465 + gnupg2
Hi python-apt folks--
On Thu 2016-08-25 20:55:27 -0400, Chris Lamb wrote:
> Source: python-apt
> Version: 1.1.0~beta4
> Severity: serious
> Justification: fails to build from source
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: ftbfs
> X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
>
> Dear Maintainer,
>
> python-apt fails to build from source in unstable/amd64:
>
> set -e; for python in python2.7 python3.5 ; do \
> $python tests/test_all.py -q || [ "linux" = "hurd" ]; \
> done;
> Warning: apt-key output should not be parsed (stdout is not a terminal)
> Warning: apt-key output should not be parsed (stdout is not a terminal)
> Warning: apt-key output should not be parsed (stdout is not a terminal)
> Warning: apt-key output should not be parsed (stdout is not a terminal)
> Warning: apt-key output should not be parsed (stdout is not a terminal)
> Warning: apt-key output should not be parsed (stdout is not a terminal)
this warning is advice about one of the things that might be going wrong
in some places in the test suite ;) I see that in upstream
7a9a292fd604bc164eed3d3fee1dc9167141d88c you're deprecating this, but i
strongly caution against this.
> [tests] Running on 2.7.12+ (default, Aug 4 2016, 20:04:34) [GCC 6.1.1
> 20160724]
> Using library_dir:
> '/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/build/lib.linux-x86_64-2.7'WARNING:
> Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> WARNING: Failed to read mirror file
> ==
> ERROR: testAddKeyFromServer (test_auth.TestAuthKeys)
> Install a GnuPG key from a remote server.
> --
> Traceback (most recent call last):
> File
> "/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/tests/test_auth.py",
> line 220, in testAddKeyFromServer
> "hkp://localhost:%d" % self.keyserver_port)
> File
> "/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/build/lib.linux-x86_64-2.7/apt/auth.py",
> line 128, in add_key_from_keyserver
> _add_key_from_keyserver(keyid, keyserver, tmp_keyring_dir)
> File
> "/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/build/lib.linux-x86_64-2.7/apt/auth.py",
> line 156, in _add_key_from_keyserver
> keyserver, keyid))
> AptKeyError: recv from 'hkp://localhost:19191' failed for
> '0xa1bD8E9D78F7FE5C3E65D8AF8B48AD6246925553'
apt/auth.py appears to want to force gnupg to store its secret key
material in secring.gpg. This isn't a best practice, and modern
versions of gpg do not do so by default. I'd recommend dropping
tmp_secret_keyring entirely.
furthermore, recent versions of gnupg (>= 2.1) do not fetch things from
keyservers directly -- if you want modern gpg to talk to the network,
you'll need to ensure that dirmngr is installed.
If you prefer to keep this test intact, you might want to build-depend
on dirmngr.
> ==
> FAIL: testAddAndExportKey (test_auth.TestAuthKeys)
> Add an example key.
> --
> Traceback (most recent call last):
> File
> "/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/tests/test_auth.py",
> line 157, in testAddAndExportKey
> WHEEZY_KEY.split("\n")[2:])
> AssertionError: Lists differ: ['mQINBE+a7rUBEADQiEKtLOgqiq8Y... != ['',
> 'mQINBE+a7rUBEADQiEKtLOgq...
>
> First differing element 0:
> 'mQINBE+a7rUBEADQiEKtLOgqiq8YY/p7IFODMqGPR+o1vtXaksie8iTOh3Vxab38'
> ''
>
> Second list contains 1 additional elements.
> First extra element 81:
> '-END PGP PUBLIC KEY BLOCK-'
>
> Diff is 5698 characters long. Set self.maxDiff to None to see it.
This change is due to the fact that upstream has stopped emitting the
Version: pseudoheader at all as of 2.1.14. Perhaps your build-dependency on
gnupg should be (>= 2.1.14) and you should strike the Version:
pseudoheader in WHEEZY_KEY in tests/test_auth.py
> ==
> FAIL: testAddAndListKey (test_auth.TestAuthKeys)
> Add an example key and test if it is correctly returned by
> --
> Traceback (most recent call last):
> File
> "/home/lamby/temp/cdt.20160826014142.YuAeJNcC8b.db.python-apt/python-apt-1.1.0~beta4/tests/test_auth.py",
> line 168, in testAddAndListKey
> "Debian Archive Automatic S
Re: Bug#835592: php-crypt-gpg: FTBFS: Tests: 286, Assertions: 514, Errors: 77, Failures: 8, Skipped: 8
Control: affects 835592 src:gnupg2 Hi php maintainers-- over on https://bugs.debian.org/835592 , Chris Lamb wrote: > Source: php-crypt-gpg > Version: 1.4.1-1 > Severity: serious > Justification: fails to build from source > User: reproducible-builds@lists.alioth.debian.org > Usertags: ftbfs > X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org > > Dear Maintainer, > > php-crypt-gpg fails to build from source in unstable/amd64: > > [..] > > +'_hasPrivate' => false > > @@ @@ >'_usage' => 1 > -'_hasPrivate' => true > +'_hasPrivate' => false >'_isRevoked' => false >) >) >) >) > > > /home/lamby/temp/cdt.20160827100527.NHNbg5FUA3.db.php-crypt-gpg/php-crypt-gpg-1.4.1/Crypt_GPG-1.4.1/tests/GetKeysTest.php:297 > > 7) KeyGeneratorTestCase::testSetExpirationDate_string > Setting expiration date by string failed. > Failed asserting that 196400 matches expected 20. > > > /home/lamby/temp/cdt.20160827100527.NHNbg5FUA3.db.php-crypt-gpg/php-crypt-gpg-1.4.1/Crypt_GPG-1.4.1/tests/KeyGeneratorTest.php:240 > > 8) SubKeyTestCase::testParseCreationDateIso > Failed asserting that two objects are equal. > --- Expected > +++ Actual > @@ @@ >Crypt_GPG_SubKey Object ( >'_id' => '8C37DBD2A01B7976' >'_algorithm' => 16 >'_fingerprint' => '' >'_length' => 2048 > -'_creationDate' => 1221442255 > +'_creationDate' => 1221438655 >'_expirationDate' => 0 >'_usage' => 1 >'_hasPrivate' => false >'_isRevoked' => false >) > > > /home/lamby/temp/cdt.20160827100527.NHNbg5FUA3.db.php-crypt-gpg/php-crypt-gpg-1.4.1/Crypt_GPG-1.4.1/tests/SubKeyTest.php:209 > > ERRORS! > Tests: 286, Assertions: 514, Errors: 77, Failures: 8, Skipped: 8. > debian/rules:7: recipe for target 'override_dh_auto_test' failed > make[1]: *** [override_dh_auto_test] Error 2 > make[1]: Leaving directory > '/home/lamby/temp/cdt.20160827100527.NHNbg5FUA3.db.php-crypt-gpg/php-crypt-gpg-1.4.1' > debian/rules:3: recipe for target 'build' failed > make: *** [build] Error 2 > > [..] I suspect that many of these 77 failures are the result of php-crypt-gpg not working properly with gnupg 2.1.x. (though some of them do appear to be timestamp variations that might be worth unravelling separately) I'll be releasing a new version of gnupg shortly that will explicitly declare that it Breaks: php-crypt-gpg (<= 1.4.1-1). Ideally, the next version of php-crypt-gpg can have this bug fixed and it will work cleanly with the modern version of gnupg. However, if your next upload of php-crypt-gpg can't be built or run against modern versions of GnuPG, then you probably need to state this package's dependency on gnupg as gnupg (<= 2.1). Regards, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs
On Fri 2016-08-12 15:27:40 -0400, Thomas Schmitt wrote: > Although grub-mkrescue probably can live with poor GPT GUIDs, i meanwhile > found a use case in xorriso where user defined modification-date does not > express the desire for reproducibile GUIDs: xorriso command > -boot_image "any" "replay". > If xorriso modifies a bootable ISO made by grub-mkrescue, then it has > to maintain the modification date so that GRUB2 after waking up finds > the ISO. It is then inappropriate to keep GPT GUIDs, because the ISOs > are nevertheless not meant to be identical. > > So the default of new option --gpt_disk_guid is old behavior "random". Would it possible to generate the GPT GUID based on a digest of the contents of the ISO itself? I don't understand well enough how GPT interacts with ISOs to be able to sketch out the details, but if there is a way to look at the rest of the generated filesystem *aside* from the GUID, then you could push all that data through a simple hash function, and then deterministically derive the GUID from the hash function. (what hash function to use? it probably doesn't even need to be cryptographically secure, but sha256 is cheap these days and it avoids any risk that someone could come up with a plausible attack based on forcing GUID collisions) That would give you identical GUIDs for identical ISOs, and distinct GUIDs for ISOs that vary in any way, without having to include any randomness or asking the user to do the work to select a non-random GUID (which they're probably not likely to do responsibly). Thanks for your work on this, Thomas! Let me know if this idea doesn't make sense for some reason, like if there are other bits in the ISO that themselves depend on the GUID. I'd be happy to brainstorm other approaches. --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#833282: reprotest: please add a manpage for reprotest
Package: reprotest Version: 0.2 Severity: normal Thanks for writing reprotest! it would be great to have a manual page for the executable, since that's where many users look for documentation. it would also clear up this lintian warning: W: reprotest: binary-without-manpage usr/bin/reprotest all the best, --dkg -- System Information: Debian Release: stretch/sid APT prefers testing-debug APT policy: (500, 'testing-debug'), (500, 'testing'), (200, 'unstable-debug'), (200, 'unstable'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages reprotest depends on: ii apt-utils 1.3~pre2 ii diffoscope 56 ii libdpkg-perl1.18.9 ii procps 2:3.3.12-2 ii python3-debian 0.1.28 pn python3:any Versions of packages reprotest recommends: pn autodep8 pn disorderfs pn locales-all ii qemu-system 1:2.6+dfsg-3 ii qemu-utils 1:2.6+dfsg-3 pn schroot reprotest suggests no packages. -- no debconf information ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#833284: reprotest: please add a "buildpath" variation to reprotest
Package: reprotest Version: 0.2 Severity: normal The list of variations supported by reprotest doesn't appear to include changing the build path. We should be able to generate identical output regardless of where in the filesystem the build takes place, so supporting this variation would be a useful contribution to overall reproducibility. thanks for writing reprotest! --dkg -- System Information: Debian Release: stretch/sid APT prefers testing-debug APT policy: (500, 'testing-debug'), (500, 'testing'), (200, 'unstable-debug'), (200, 'unstable'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages reprotest depends on: ii apt-utils 1.3~pre2 ii diffoscope 56 ii libdpkg-perl1.18.9 ii procps 2:3.3.12-2 ii python3-debian 0.1.28 pn python3:any Versions of packages reprotest recommends: pn autodep8 pn disorderfs pn locales-all ii qemu-system 1:2.6+dfsg-3 ii qemu-utils 1:2.6+dfsg-3 pn schroot reprotest suggests no packages. -- no debconf information ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#827155: dpkg-buildflags: reproducible/fixdebugpath doesn't escape build path
On Sun 2016-06-12 23:25:33 -0400, HW42 wrote: > as Mattia noticed dpkg-buildflags doesn't escape the build path in the > -fdebug-prefix-map CC argument when enabling the 'fixdebugpath' option. > > What assumptions does dpkg make about the build path? I think there are a > lot of build scripts which anyway break if the build path contains a > space. that seems likely to me -- not just debian packaging scripts either, but likely upstream build toolchain decisions as well. > Unfortunately this isn't trivially fixed since the flags are > concatenated as a string and later exported in different languages > (shell, Makefile). So if we want to support spaces in buildpaths this > need to be changed to an array. What do you think? I think you mean that the flags should be internally stored as an array inside dpkg-dev instead of a concatenated string. That sounds right to me, and a worthwhile change in its own right. It won't fix any upstream toolchain bugs, of course, and if we start to vary by having whitespace (or even weirder characters) in paths, we're likely to see a large number of FTBFS. fixing dpkg-dev to treat the flags internally as an array seems like a good first step, though. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#828178: afl: FTBFS: clang error unknown argument -fdebug-prefix-map=/build/afl-2.16b=.'
Hi Daniel-- On Sat 2016-06-25 16:09:08 -0400, Daniel Stender wrote: > AFL fails to build from source in reproducible builds test build > environments like this (log from 2.16b-1 build): > > > lang-3.7 -g -O2 -fdebug-prefix-map=/build/afl-2.16b=. -fPIE > -fstack-protector-strong -Wformat -Werror=format-security -Wall > -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/lib/afl\" > -DBIN_PATH=\"/usr/bin\" -DVERSION=\"2.16b\" afl-clang-fast.c -o > ../afl-clang-fast -fPIE -pie -Wl,-z,relro -Wl,-z,now > clang: error: unknown argument: '-fdebug-prefix-map=/build/afl-2.16b=.' > Makefile:79: recipe for target '../afl-clang-fast' failed > I think clang introduced -fdebug-prefix-map in 3.8.0 (see https://bugs.debian.org/819185) and afl build-depends on clang-3.7. I assume that it's the reproducible-builds toolchain that's adding the -fdebug-prefix-map option to CFLAGS, right? That's good -- it should help avoid variations in the generated binaries due to build path alone, so please keep it! seems like the right fix here is either to build afl against a newer version of clang, or to resolve #819185 by backporting the option to clang-3.7. > I'm going in it, soon. I don't know what this means, sorry! --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [Groff] Reproducible dates in HTML/PDF/PS output files?
On Mon 2016-06-13 09:20:46 -0400, Ximin Luo wrote: > Ximin Luo: >> Hi all, was this ever committed? I don't see it in the commit archives... >> > > Hey, any news on this? The original thread I'm referring to is this one: > > https://lists.gnu.org/archive/html/groff/2015-11/msg00013.html Werner LEMBERG said "Looks good, please proceed" here: https://lists.gnu.org/archive/html/groff/2015-11/msg00038.html but i don't know whether that was directing Colin or someone else to proceed (or whether Colin is able to do so directly), and what the next step actually is. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#819194: dpkg-buildflags: please add normalizedebugpath to reproducible feature set
On Tue 2016-03-29 21:52:53 -0400, Holger Levsen wrote: > it's surely progress on the gcc/clang side of things but dropping the > build path from the .buildinfo files would be a huge step *backwards* > for reproducibility… No one is arguing for dropping the build path from .buildinfo files. As we discussed at the reproducible summit, .buildinfo files serve two purposes: 0) they document the environment used during a specific build, to a level of detail that should make it at least possible to reproduce the build. 1) they also document things that should *not* be necessary to reproduce the build, but might be under some circumstances, or for some packages. I think the build path falls into category (1) here. In an ideal scenario, we could have two buildinfo files with variations on the build environment (buildpath, minor versions of build-deps, etc) and *still* have reproducible binary outputs. This would let us know that the variations in question are not things that cause variation in the output. > Also, c/c++ packages today only make up a small portion of the archive. > Probably this famous someone should do a rebuild of the archive, using > our toolchain (and this patch), using arbitrary build pathes. yes, that would be great! --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#819194: dpkg-buildflags: please add normalizedebugpath to reproducible feature set
On Tue 2016-03-29 20:58:32 -0400, Holger Levsen wrote: > not wanting to spoil the fun, but… > > On Mon, Mar 28, 2016 at 06:33:49PM -0400, Daniel Kahn Gillmor wrote: >> > Ah great! And one less way to leak local information. >> yep :) > > I *believe* it's not enough (for reproducible builds in arbitrary > pathes) if gcc+clang can now cope. IIRC there are other compilers that > have the same behaviour, eg ocaml and erlang, but probably others too. > > Someone shoulds to check this and confirm or deny though. This isn't fun-spoiling, it's a useful reality check. But if we were required to get all the way to 100% before we made any progress, then reproducible builds wouldn't have gotten off the ground at all. The changes proposed in this bug report are a good step that should handle a very large proportion of the debian archive. The fact that there will remain corners of the archive that need additional work is fine: we can turn our attention to the remaining 20% once we get 80% of the buildpaths resolved. -dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#819194: dpkg-buildflags: please add normalizedebugpath to reproducible feature set
Package: dpkg-dev
Version: 1.18.4
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: buildpath
Compilers tend to inject the current path of the filesystem into the
debug symbols, so that the debugger can find the sourcecode. But this
isn't actually useful for packages shipped as binaries, because the
path on the build machine isn't relevant on the deployed machine.
It also causes problems for reproducible builds, because the same
source built in two different locations in the filesystem will result
in two different sets of debug symbols.
With gcc (and soon clang, see #819185), you can re-map the path stored
in the debug symbols with -fdebug-prefix-map, so it can be used to
normalize the paths stored in the debug symbols.
The attached patch adds a "normalizedebugpath" feature to the
"reproducible" feature set, which appends -fdebug-prefix-map=CWD=.
(where CWD is the actual current working directory) to CFLAGS and
CXXFLAGS. It is off by default.
Regards,
--dkg
-- System Information:
Debian Release: stretch/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'testing'), (200, 'unstable'), (1,
'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages dpkg-dev depends on:
ii base-files9.6
ii binutils 2.26-5
ii bzip2 1.0.6-8
ii libdpkg-perl 1.18.4
ii make 4.1-9
ii patch 2.7.5-1
ii xz-utils 5.1.1alpha+20120614-2.1
Versions of packages dpkg-dev recommends:
ii build-essential 11.7
ii clang-3.5 [c-compiler] 1:3.5.2-3
ii clang-3.6 [c-compiler] 1:3.6.2-3
ii clang-3.7 [c-compiler] 1:3.7.1-2
ii fakeroot 1.20.2-1
ii gcc [c-compiler] 4:5.3.1-1
ii gcc-5 [c-compiler] 5.3.1-12
ii gnupg1.4.20-4
ii gnupg2 2.1.11-6
ii gpgv 1.4.20-4
ii gpgv22.1.11-6
pn libalgorithm-merge-perl
ii tcc [c-compiler] 0.9.27~git20151227.933c223-1
Versions of packages dpkg-dev suggests:
ii debian-keyring 2016.03.22
-- debconf-show failed
>From 638a575180174df9bd1e60a8856609ba72d52849 Mon Sep 17 00:00:00 2001
From: Daniel Kahn Gillmor
Date: Thu, 24 Mar 2016 13:19:28 -0400
Subject: [PATCH] add normalizedebugpath to reproducible featureset
This feature normalizes the path stored in debug symbols, so that
these symbols can be built reproducibly regardless of the location of
the build in the larger filesystem.
It defaults to off, but should be enabled by systems trying to
generate reproducible packages.
---
man/dpkg-buildflags.1 | 8
scripts/Dpkg/Vendor/Debian.pm | 9 +
2 files changed, 17 insertions(+)
diff --git a/man/dpkg-buildflags.1 b/man/dpkg-buildflags.1
index c86b01a..7cd6050 100644
--- a/man/dpkg-buildflags.1
+++ b/man/dpkg-buildflags.1
@@ -375,6 +375,14 @@ to \fBCPPFLAGS\fP.
This will cause warnings when the \fB__TIME__\fP, \fB__DATE__\fP and
\fB\%__TIMESTAMP__\fP macros are used.
.
+.TP
+.B normalizedebugpath
+This setting (disabled by default) adds
+.B \-fdebug\-path\-map=BUILDPATH=.
+to \fBCFLAGS\fP and \fBCXXFLAGS\fP, where \fBBUILDPATH\fP is set to
+the top-level directory of the package being built. This has the
+effect of removing the build path from any generated debug symbols.
+.
.SH ENVIRONMENT
There are 2 sets of environment variables doing the same operations, the
first one (DEB_\fIflag\fP_\fIop\fP) should never be used within
diff --git a/scripts/Dpkg/Vendor/Debian.pm b/scripts/Dpkg/Vendor/Debian.pm
index bf47144..ff76959 100644
--- a/scripts/Dpkg/Vendor/Debian.pm
+++ b/scripts/Dpkg/Vendor/Debian.pm
@@ -30,6 +30,7 @@ use Dpkg::ErrorHandling;
use Dpkg::Control::Types;
use Dpkg::BuildOptions;
use Dpkg::Arch qw(get_host_arch debarch_to_debtriplet);
+use Cwd;
use parent qw(Dpkg::Vendor::Default);
@@ -158,6 +159,7 @@ sub _add_reproducible_flags {
# Default feature states.
my %use_feature = (
timeless => 1,
+normalizedebugpath => 0,
);
# Adjust features based on user or maintainer's desires.
@@ -168,6 +170,13 @@ sub _add_reproducible_flags {
$flags->append('CPPFLAGS', '-Wdate-time');
}
+# Avoid storing the build path in the debug symbols
+if ($use_feature{normalizedebugpath}) {
+ my $map = '-fdebug-prefix-map=' . cwd() . '=.';
+ $flags->append('CFLAGS', $map);
+ $flags->append('CXXFLAGS', $map);
+}
+
# Store the feature usage.
while (my ($feature, $enabled) = each %use_feature) {
$flags->set_feature('reproduci
Re: [Reproducible-builds] Bug#812428: libgcrypt20: please make the build reproducible (timestamps)
On Sat 2016-01-23 16:52:09 -0500, Jérémy Bobbio wrote: > While working on the “reproducible builds” effort [1], we have noticed > that libgcrypt20 could not be built reproducibly. > > The attached patch adds support for SOURCE_DATE_EPOCH to set the value > of BUILD_TIMESTAMP defined in the configure script. Once applied, > libgcrypt20 is nearly reproducible in our current experimental > framework—their might be remaining timestamps in the PDF documentation. Thanks for fixing this, Lunar. fwiw, libgcrypt upstream is the same folks as libgpg-error upstream. In response to concerns about reproducible builds, libgpg-error 1.20 switched to not embedding the build timestamp at all upstream by default. Perhaps they'd be open to the same approach for libgcrypt if it were proposed. > --- a/debian/rules > +++ b/debian/rules > @@ -48,4 +48,4 @@ override_dh_gencontrol: > dh_gencontrol --remaining-packages > > %: > - dh $@ --parallel --with autotools_dev > + dh $@ --parallel --with autoreconf --with autotools_dev /usr/share/doc/autotools-dev/README.Debian.gz says: >> Do NOT use dh-autoreconf and dh_autotools-dev_* helpers at the same >> time, dh-autoreconf takes care of updating config.sub and >> config.guess by itself. Regards, --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] build paths and debug info for generated C objects
On Thu 2015-12-10 12:38:56 -0500, Niels Thykier wrote: > Given no one has voiced their concerns, I think you should go ahead and > push it upstream. :) I've done so, and it's on https://gcc.gnu.org/ml/gcc-patches/2015-12/msg01168.html (also forwarded to rb-gene...@reproducible-builds.org, since there's nothing debian-specific about it). --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] build paths and debug info for generated C objects
Hey all--
Niels, anthraxx and i were just commiserating about the fact that we're
punting on reproducibility of the build path. We think we might have
found a way to make progress on this.
Problem Statement
-
One of the main concerns about the build path is that it gets included
by gcc in any generated DWARF [0] debugging symbols, specifically in the
dwarf attribute named DW_AT_comp_dir.
Background
--
gcc already allows the user to tweak this attribute directly:
-fdebug-prefix-map=old=new
When compiling files in directory old, record debugging information
describing them as in new instead.
So, for example, i can do:
gcc -fdebug-prefix-map=$(pwd)=. -o test test.c
gdb still works for me when debugging code that is built this way.
However, gcc also stores all the switches used during the build in the
DW_AT_producer, so if you do this, then you're just moving the build
path to a different dwarf attribute, so it's still being encoded in the
output. This doesn't solve the reproducibility problem, but it provides
us with a way to demonstrate that removing the data from DW_AT_comp_dir
doesn't cripple our ability to debug.
We also observed that DW_AT_name already stores the name of the compiled
file relative to the DW_AT_comp_dir -- this poses no reproducibility
problems on its own.
Proposed Solution
-
A minor change to gcc:
* if the "old" parameter for -fdebug-prefix-map starts with a literal $
character, make gcc treat it as an environment variable name. So:
(note the shell escaping)
export SOURCE_BUILD_DIR=$(pwd)
gcc -g -fdebug-prefix-map=\$SOURCE_BUILD_DIR=. -o test.o -c test.c
should do what we need: the gcc flags are static, and the build path
is stripped.
- What to do if the chosen env var isn't set? Probably just skip the
match entirely, and maybe raise a warning.
- What about bizarre theoretical filesystems that might have $ as a
leading character? We don't know of any. We're willing to
sacrifice them for this feature. :)
I've patched GCC to work this way successfully:
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 5256031..234432f 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -6440,7 +6440,9 @@ link processing time. Merging is enabled by default.
@item -fdebug-prefix-map=@var{old}=@var{new}
@opindex fdebug-prefix-map
When compiling files in directory @file{@var{old}}, record debugging
-information describing them as in @file{@var{new}} instead.
+information describing them as in @file{@var{new}} instead. If
+@file{@var{old}} starts with a @samp{$}, the corresponding environment
+variable will be dereferenced, and its value will be used instead.
@item -fno-dwarf2-cfi-asm
@opindex fdwarf2-cfi-asm
diff --git a/gcc/final.c b/gcc/final.c
index 8cb5533..bc43b61 100644
--- a/gcc/final.c
+++ b/gcc/final.c
@@ -1525,6 +1525,9 @@ add_debug_prefix_map (const char *arg)
{
debug_prefix_map *map;
const char *p;
+ char *env;
+ const char *old;
+ size_t oldlen;
p = strchr (arg, '=');
if (!p)
@@ -1532,9 +1535,29 @@ add_debug_prefix_map (const char *arg)
error ("invalid argument %qs to -fdebug-prefix-map", arg);
return;
}
+ if (*arg == '$')
+{
+ env = xstrndup (arg+1, p - (arg+1));
+ old = getenv(env);
+ if (!old)
+ {
+ warning (0, "environment variable %qs not set in argument to "
+ "-fdebug-prefix-map", env);
+ free(env);
+ return;
+ }
+ oldlen = strlen(old);
+ free(env);
+}
+ else
+{
+ old = xstrndup (arg, p - arg);
+ oldlen = p - arg;
+}
+
map = XNEW (debug_prefix_map);
- map->old_prefix = xstrndup (arg, p - arg);
- map->old_len = p - arg;
+ map->old_prefix = old;
+ map->old_len = oldlen;
p++;
map->new_prefix = xstrdup (p);
map->new_len = strlen (p);
What do r-b people think about this? I'm happy to try to push this
patch to the gcc upstream if folks here think this sounds reasonable and
would address a real future r-b issue.
Alternate Solutions
---
We considered and discarded several other possible solutions, which i'm
noting below, along with the downsides that led us to select the one we
chose:
* ask gcc to not record -fdebug-prefix-map options in DW_AT_producer
- it's weird that some options wouldn't be recorded and some
would.
- build systems would need to set dynamic CFLAGS not be able to
use this approach. debian can do this in dpkg-buildpackage, but
apparently it's tougher on Arch (though Arch can more easily
set dynamic environment variables).
or three different ideas for new gcc flags, all of which share the
problem that adding a new gcc option would mean that attempts to apply
this prefix map would fail hard when using non-updated gcc:
* gcc -fdebug-prefix-map-from-env=NEW
This evaluates a specific, fixed environment variable like
SOURCE_BUILD_ROOT
Re: [Reproducible-builds] Bug#763822: ftp.debian.org: please include .buildinfo file in the archive
Hi there! In https://bugs.debian.org/763822, lunar asked ftp.debian.org to accept .buildinfo files when they are uploaded with a .changes file. This is a followup to make the request concrete by specifying how we hope the archive will sanity-check the included .buildinfo files, and with a suggestion of how they could be distributed across the mirrors in a way that will be reasonably convenient for users and downstreams without making mirror operators crazy. I'm writing this after discussions with Jelmer, Niels, Lunar, and others involved in the Reproducible Builds project. Constraints guiding the suggestions below - We want an archive user to be able to find and fetch all .buildinfo files that produced a given binary package We want the eventual possibility of multiple .buildinfo files per We understsand that mirror operators don't like small files because rsync gets fussy with them. We want both buildds and debian developers to be able to upload .buildinfo files. Asks of ftp-master -- We hope that the archive will verify .buildinfo files uploaded by buildds and DDs or DMs. We don't expect to require buildds or DDs or DMs to upload .buildinfo files at this time, though we hope they'll start to do so once the archive can accept them. Here's how we think the archive might sanity-check them: * There may be 0 or more .buildinfo files included in a .changes file. Each .buildinfo file describes an environment that was used to produce some of the binary artifacts (e.g. .deb, .udeb, etc) in this upload. * To validate each .buildinfo file: * ensure that the filename is of the form __.buildinfo where: * matches the source name in the Source: field * equals the Version: field * is /[-a-z0-9]+/ * ensure that this filename is not already in the archive. * the file should be clearsigned OpenPGP in UTF-8, with nothing outside the OpenPGP framing. It should have a valid signature from the same OpenPGP key that signed the .changes file. * The signed part of the file must be a valid control file. * ensure that Source: and Version: fields in the .buildinfo matches the Source: and Version: fields in the .changes file. * the .buildinfo must include the same .dsc as the .changes file, with the same checksum. * in addition, the .buildinfo file should list at least one binary artifact. * for every binary artifact listed in the .buildinfo file: * ensure that it is listed in the .changes file with the same checksum(s). (fwiw, if anyone is concerned about minimizing the size of the .buildinfo file, there is no need to include the md5 or sha1 checksums of the artifacts. The Checksums-Sha256: sub-stanza is sufficient for our purposes) If an included .buildinfo file doesn't validate, please reject the entire upload. Once an upload that includes some .buildinfo files is accepted, we want users to be able to find the .buildinfo(s) for a binary package if they want to try to reproduce it. Here's a concrete suggestion for how to do that in a way that might not make mirror operators sad (if you have a different or preferred suggestion, that'd be great too): * collect all .buildinfo files in the archive that produced binary artifacts for a given architecture in a tarball named Buildinfos.tgz which is distributed alongside Packages. (for example, binary-amd64/Buildinfos.tgz and binary-all/Buildinfos.tgz). * the structure of the tarball could be //__.buildinfo (with the same expansions as above) * the gzip layer of the tarball should be --rsyncable * When re-creating the Buildinfos.tgz file after some binary artifacts have been removed from a suite, any .buildinfo file that only references artifacts no longer in the suite can be removed. Does this seem like a reasonable way to distribute this information? Clarifications from original bug report --- In the time since this bug was originally posted, we have a clearer understanding of what a .buildinfo file represents, and how it can be used. For clarity and future documentation, i'll amend/nitpick a few comments from the original text of the bug report below: > .buildinfo files would capture from the build environment as much > information as needed to reproduce the build. The .buildinfo file *may* contain more information than is needed to reproduce the build. The goal is to have it provide enough of a record of the build environment to be able to reproduce the build, but it's also possible to include additional, unneeded information, and that's OK. (e.g. we are likely to include the exact version number of some essential packages, even though going from coreutils 8.17-1 to 8.17-2 is unlikely to affect the build). > * A .buildinfo file is generated for each
[Reproducible-builds] [R-B WEBSITE PATCH] native en_US speaker website tweaks
Signed-off-by: Daniel Kahn Gillmor --- index.html | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/index.html b/index.html index c3808c9..b361343 100644 --- a/index.html +++ b/index.html @@ -37,13 +37,13 @@ layout: home - Most aspect of software verification is done on source code, as that is + Most aspects of software verification are done on source code, as that is what humans can reasonably understand. But most of the time, computers require software to be first built into a long string of numbers to be used. With reproducible builds, multiple parties can redo this process independently and ensure they all get exactly the same result. We can thus - grow confidence that a distributed binary code is indeed + gain confidence that a distributed binary code is indeed coming from a given source code. @@ -53,7 +53,7 @@ layout: home about its sensors in a lab environment. Having the source code under public scrutiny would have made adding such a misfeature only a little more difficult. Without reproducible builds, it is hard to - attest than the binary code running in the car was actually made using + confirm that the binary code installed in the car was actually made using the source code that has been verified. @@ -84,7 +84,7 @@ layout: home Third, users should be given a way to recreate a close enough build environment, perform the build process, and verify that the output - match the original build. + matches the original build. Learn more on how to make your software build -- 2.6.2 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#802528: valac: make the generated C files reproducible
On Tue 2015-10-20 15:53:43 -0400, Niko Tyni wrote:
> Package: valac
> Version: 0.30.0-2
> Severity: wishlist
> Tags: patch
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: toolchain randomness
> X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
>
> While working on the "reproducible builds" effort [1], we have noticed
> that the gnome-clocks package doesn't build reproducibly because the
> C files generated by valac vary between builds. A sample diff is
>
> --- a/src/timer.c
> +++ b/src/timer.c
> @@ -1277,8 +1277,8 @@ static void
> clocks_timer_face_clocks_clock_interface_init (ClocksClockIface * if
>static void clocks_timer_face_instance_init (ClocksTimerFace * self) {
> self->priv = CLOCKS_TIMER_FACE_GET_PRIVATE (self);
> self->priv->_state = CLOCKS_TIMER_FACE_STATE_STOPPED;
> - g_type_ensure (CLOCKS_TYPE_ANALOG_FRAME);
> g_type_ensure (CLOCKS_TIMER_TYPE_COUNTDOWN_FRAME);
> + g_type_ensure (CLOCKS_TYPE_ANALOG_FRAME);
> gtk_widget_init_template (GTK_WIDGET (self));
>}
>
> The order of the two g_type_ensure() calls varies more or less randomly.
>
> I've tracked this down to a HashSet data structure in Vala.GtkModule
> that holds these classes in the C code generation phase. Changing that
> to an ordered List type instead fixes the issue. Please consider the
> attached patch.
Thanks for catching this! I'd looked into it before and hadn't been
able to track this down.
But doesn't the change from HashSet to List end up changing the behavior
with regard to multiple children of the same class?
That is, if it's a List, then the same class can appear multiple times,
whereas if it's a set, i think it just gets added once.
The usual approach would be to sort the output where it's produced from
the HashedSet, and not to change the data structure itself to allow
duplicates.
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [Debian Wiki] Update of "ReproducibleBuilds/TimestampsProposal" by BenBoeckel
On Wed 2015-09-16 21:19:59 -0400, Debian Wiki wrote:
> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Debian Wiki" for
> change notification.
>
> The "ReproducibleBuilds/TimestampsProposal" page has been changed by
> BenBoeckel:
> https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal?action=diff&rev1=44&rev2=45
>
> Comment:
> CMake: update example to use lower-case command names, use
> $ENV{SOURCE_DATE_EPOCH}, fix argument quoting, strip trailing whitespace, and
> formatting cleanup so it is easier to read
>
>
> {{{
> if (DEFINED ENV{SOURCE_DATE_EPOCH})
> - EXECUTE_PROCESS(COMMAND "date" "-u" "-d @${SOURCE_DATE_EPOCH}"
> "+%d/%m/%Y" OUTPUT_VARIABLE BUILD_DATE)
> + execute_process(
> + COMMAND "date" "-u" "-d" "@$ENV{SOURCE_DATE_EPOCH}" "+%d/%m/%Y"
> + OUTPUT_VARIABLE BUILD_DATE
> + OUTPUT_STRIP_TRAILING_WHITESPACE)
> - else()
> + else ()
> - EXECUTE_PROCESS(COMMAND "date" "+%d/%m/%Y" OUTPUT_VARIABLE BUILD_DATE)
> + execute_process(
> + COMMAND "date" "+%d/%m/%Y"
> + OUTPUT_VARIABLE BUILD_DATE
> + OUTPUT_STRIP_TRAILING_WHITESPACE)
> - endif()
> + endif ()
> }}}
>
> ''The above will work only with GNU date. See POSIX shell example on how to
> support BSD date.''
Is there a reason above that this example code to translate from
SOURCE_DATE_EPOCH to a build date doesn't produce ISO-8601-compliant
date strings in the BUILD_DATE variable?
I'm happy to replace "+%d/%m/%Y" with "+%Y-%m-%d", but i don't know much
about cmake and don't want to get in an edit war about this.
fwiw, "%d/%m/%Y" may look sane to most people around the world, but it
has the following disadvantages over ISO-8601:
0) lexicographically-sorted dates are not chronologically-sorted
1) the US typically uses "+%m/%d/%Y" (which is insane, granted) which
means that 06/09/2015 is easily interpretable as June 9th as well as
September 6th.
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [PATCH 2/2] DocBook: Use a fixed encoding for output
On Fri 2015-09-11 15:30:59 -0400, Jonathan Corbet wrote: > On Tue, 01 Sep 2015 23:49:19 +0100 > Ben Hutchings wrote: > >> Currently the encoding of documents generated by DocBook depends on >> the current locale. Make the output reproducible independently of >> the locale, by setting the encoding to UTF-8 (LC_CTYPE=C.UTF-8) by >> preference, or ASCII (LC_CTYPE=C) as a fallback. > > I guess I have to ask, though: doesn't it seem that having the docs > produced according to the current locale is the Right Thing to do? Users > have their locale set as it is for a reason, it seems like the production > of textual documents should respect their choice. > > Am I missing something here? I sympathize with Jonathan's general concern here -- if this patchset makes it impossible for people to build documentation with (for example) their preferred collation order, it would be suboptimal. On the other hand, this seems to focus on character encodings specifically; do we really want to encourage any sort of encodings other than UTF-8? The only plausible arguments i've heard for documents that are exclusively CJK characters, which could achieve a modest size reduction using more targeted encodings. afaik, there are no such documents in the kernel, and i doubt there ever will be. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#793813: medicalterms: /usr/share/hunspell/de_med.dic is not reproducible (collation order based on build environment)
Source: medicalterms Version: 20110608-1 Severity: wishlist User: reproducible-builds@lists.alioth.debian.org Usertags: locale Hi medicalterms maintainers-- Today i noticed that usr/share/hunspell/de_med.dic gets created differently depending on the locale of the build environment: https://reproducible.debian.net/rb-pkg/unstable/amd64/medicalterms.html I suspect that the generation of this package should explicitly set the locale in question, even if LC_ALL is set by the environment. i see that Makefile.am has LANG=de_DE on line 85, but locale(7) suggests that LC_ALL would take precedence over LANG during collation. Regards, --dkg -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (500, 'testing'), (200, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.0.0-2-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] comparing PE executables and assemblies in debbindiff
hi folks-- I just pushed git commit 3d0af25 to the debbindiff repository: commit 3d0af25e8d046b9ae08405cdbf243dc38ae543b4 Author: Daniel Kahn Gillmor Date: Sun Jul 12 17:10:13 2015 -0400 add test for PE assemblies and executables This works at least for those assemblies generated by mono. pedump does not currently provide useful output for PE binaries generated by i686-w64-mingw32-gcc, unfortunately. But this patch is a start that should help us express problems with mono builds more clearly at least. I've tested it locally against gmime (a package i'm responsible for that uses mono), and it provides nicer output for me there than the previous hexdump. I pushed it directly rather than sending it as a patch for the mailing list because it contains two binary test executables (test1.exe and test2.exe), and i don't know how they translate into a patch on the mailing list. This is my first patch to debbindiff, though, so i'm a bit more unsure of it than i would like to be. As with any patch i've authored, i welcome review and criticism if you see a way it could be better. Happy hacking, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#791673: doc-debian: please make .txt files UTF-8 independently of the locale of the build system
Package: doc-debian Version: 6.3 Severity: wishlist Tags: patch User: reproducible-builds@lists.alioth.debian.org currently, lynx produces text output on the basis of the locale. this means that as the locale differs, the generated .txt files change. Please see: https://reproducible.debian.net/dbd/unstable/amd64/doc-debian_6.3.debbindiff.html I believe the attached patch should resolve the problem. --dkg -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (500, 'testing'), (200, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Index: doc/Makefile === --- doc/Makefile (revision 10970) +++ doc/Makefile (working copy) @@ -40,7 +40,7 @@ wml -i bug-pkgreport-opts.inc -q $< >$@ %.txt: %.html - lynx -dump -nolist $< >$@ + LC_ALL=C.UTF-8 lynx -dump -nolist $< >$@ # These rules are conditioned to the existence of $(WEBWML) # so that the package can be built regardless of its existence ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Storing .deb checksums in ADMINDIR/status?
On Tue 2015-06-23 03:31:05 -0400, Jérémy Bobbio wrote: > Some people suggested that we should record a checksum of the `.deb` > installed as a way to unambiguously referring to a specific package. > The main benefit that I can think of is that it would allow to directly > retrieve the file from snapshot.debian.org based on the hash [2]. I like the idea of storing a cryptographically-strong digest of each installed package. I'm no expert on package management, but dpkg does sound to me like the right place to keep this record, for whatever that's worth. > [2]: https://anonscm.debian.org/cgit/mirror/snapshot.debian.org.git/plain/API > URL: /file/ This API is a little weird in that it doesn't specify the hash algorithm. Their examples are all 160-bits, hex-encoded, which makes me suspect that they're using SHA1. While SHA1 isn't completely practically broken yet, it's probably not a good idea to rely on it in situations like this that depend on the digest mechanism's collision-resistance over binary objects. We haven't seen a forced SHA-1 collision in published research yet, but it's just a matter of time (and we don't know what the SHA-1 collision attacks look like in private research). A stronger digest from the SHA2 family (SHA-256 or SHA-512) would be preferable if we're hardcoding the choice of digest in this first implementation. allowing for algorithm agility (hash selection at runtime) is another option, but it seems like extra engineering work. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#788536: icont embeds some entropy in produced files -- please provide a reproducible mode
Package: icon Version: 9.4.3-4.2 X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org User: reproducible-builds@lists.alioth.debian.org Usertags: toolchain Severity: wishlist Control: affects -1 noweb I'm working on the reproducible-builds project [0] the noweb package uses icont to create several executables in /usr/bin and /usr/lib/noweb. Those executables appear to have some variance depending on the build environment. The differences show up consistently between octets 0x130 and 0x230. You can see them here: https://reproducible.debian.net/dbd/unstable/amd64/noweb_2.11b-9.debbindiff.html I'm not sure what the variance comes from (timestamps? something else?), but these file are generated with icont from files in src/icon/ You can see the build log here: https://reproducible.debian.net/rbuild/unstable/amd64/noweb_2.11b-9.rbuild.log does icon have a deterministic mode that can be used for noweb? --dkg [0] https://wiki.debian.org/ReproducibleBuilds ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] cheers to ikiwiki
hey Reproducible Build folks--
Just wanted to give a shout out to Simon McVittie (cc'ed here), who did
a bunch of work getting ikiwiki to support a deterministic option.
In ikiwiki version 3.20150610, we see the following
reproducibility-oriented changes (in addition to a small fix i supplied
for #785738):
* Populate pagectime from either mtime or inode change time,
whichever is older, again for more reproducible builds
* debian: build the docwiki with LC_ALL=C.UTF-8 and TZ=UTC
* brokenlinks: sort the pages that link to the missing page, for
better reproducibility
* Add [[!meta date]] to news items and tips, since the git checkout
and build process can leave the checkout date in the tarball
release, leading to unstable sorting
* Sort backlinks deterministically, by falling back to sorting by href
if the link text is identical
* Add a $config{deterministic} option and use it for the docwiki
* haiku: if deterministic build is requested, return a hard-coded haiku
* polygen: if deterministic build is requested, use a well-known random
seed
Thanks to Simon for all this thoughtful tuning of a rather intricate
package. It's much appreciated!
Regards,
--dkg
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#787444: Bug#787444: help2man: support externally-supplied --date for reproducibility
On Fri 2015-06-05 10:55:34 -0400, Brendan O'Dea wrote: > Any of UTC_SOURCE_DATE, SOURCE_DATE_UTC My vote is for SOURCE_DATE_UTC, and i agree with Brendan that we should take the opportunity to define this as strictly and narrowly as possible (i.e. end in a 'Z', none of the other offsets), so that people relying on it know they're getting a fixed thing, and don't have to implement any fancy parsing/offsetting code if they're not already using an ISO8601-compliant date-parsing library. > REPRODUCABLE_BUILD_SOURCE_DATE_UTC would work for me I don't like this one, because apparently it's too hard to spell :) --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] generating reproducible ISOs with xorriso
Hi Thomas--
Thanks for all your work looking into this!
On Fri 2015-06-05 10:57:38 -0400, Thomas Schmitt wrote:
> About the --sort-weight-list approach which is possible with
> already released xorriso versions:
>
>> (find . -type f -print0 | xargs -0 md5sum | sort | cut -f2- -d/ ; find .
>> -mindepth 1 \! -type f | sort | cut -f2- -d/ ) | awk '{ N=N+1; print N " "
>> $0 }'
>
> I misunderstood the role of md5sum here. Actually it seems
> surplus. Why not just sort the paths ? That would be enough to
> give awk a reproducible input sequence.
Right, but it would seem to fail for hardlinked files or deduped files,
because it would weight one of the files in different places than the
other.
> Ok. The risk of a random collision is avoided and 2 billion
> files is not a severe limitation. (But the hardlinks ...)
>
> xorriso will not understand the "\n" which md5sum substitutes
> for newline characters in filenames. So trying to process such
> filenames will not be reliably reproducible and throw errors:
> xorriso : FAILURE : Cannot find path 'a\nb' in loaded ISO image
> One would have to set before -as mkisofs:
> -abort_on fatal
> in order to avoid a premature end of the program run.
> The attribution of weights would stop in any case.
Yep, i understand this limitation. For this first-pass hackery, I think
i'm ok with the idea that reproducibility fails if you put a newline in
a filename. Presumably the same goes for files that have a literal
backslash (\) in their name as well, since md5sum has to escape those
too. (sane people shouldn't be putting newlines and backslashes in
filenames anyway!)
> There is no need to attribute weight to directories.
> It applies only to the content source objects of regular files.
> ("Regular file" in the ISO, not necessarily on hard disk).
Thanks, that's useful to know, and it makes the command cleaner.
> So how about this:
>
>if test $(find . -name '*'$'\n''*' | wc -w) -gt 0
>then
> echo "FOUND FILENAMES WITH NEWLINES UNDERNEATH $(pwd)" >&2
> exit 1
>fi
>
>find . -type f -print | \
> sort | cut -f2- -d/ | awk '{ N=N+1; print N " " $0 }'
As i mentioned above, i don't like that sorting just by name seems to
miss out on the dedup/hardlink compression. I want reproducible images
*and* compact images (and a pony! :) )
Also, the above doesn't do anything for non-directory, non-regular files
(sockets, fifos, device nodes, etc) -- do those even make sense in
ISO-9660? Do we need to worry about how/where they sort?
> Extent location of regular files:
>
> The question was:
> If i sort the hardlink-merged IsoFileSrc according to
> a ISO 9660 directory tree traversal, will the sequence be
> reproducible for trees with identical file names and
> attributes ?
>
> I now verified that the directories get sorted according
> to their ISO 9660 names. The process of name collision
> resolution (mangling) is complicated but depends only on
> the user defined input names and their sequence. Name sorting
> happens before mangling and afterwards.
> (libisofs/ecma119_tree.c funtions ecma119_tree_create(),
> sort_tree(), mangle_tree(), qsort(3) in mangle_single_dir())
> So there should be no permutations of identical name lists
> possible.
This is a triply-nice result, esp. because
* it includes the hardlink-merged files, and
* it puts the extents in an order that seems intelligible from a scan
of the dirtree, and
* it piggybacks off of sorting work that's already being done, so
doesn't seem to introduce much extra overhead.
> Extent location of directories:
>
> Looks already reproducible.
> They get stored after volume descriptors but not before block 32.
> (The extent address of the root directory can be read as little
> endian 32 bit number from byte 32924 to 32927 of the ISO.
> ECMA-119 8.4.18 and 9.1.3)
> The production of extents traverses the sorted ISO tree.
> (libisofs/ecma119.c function write_dirs())
> The size of a directory extent depends on name lengths and
> attributes of the files inside the directory.
>
> Then there are the Path Tables (nobody reads them):
>
> Looks already reproducible.
> The sequence of entries is determined by an array pathlist[]
> which gets filled by traversal of the sorted ISO tree.
> (libisofs/ecma119.c function write_path_tables())
Nice, these are both good news.
> So i will go for the reproducible array of IsoFileSrc in
> libisofs/filesrc.c function filesrc_writer_pre_compute().
> The red-black tree shall merge hardlinks but not define
> the sequence of data file extents.
One other possible approach occurred to me yesterday:
What if you kept the red/black tree implementation, but keyed it by file
content digest (md5, sha1, sha256, whatever) instead of by dev/inode
tuple? Using such a red-black tree for extent placement would give you
not only hardlink discovery and reproducibility, but also automatic
deduplication for even more compact images in some situations.
Advantages:
* even more
Re: [Reproducible-builds] Bug#787793: libisoburn: new upstream version 1.4.0 available
On Fri 2015-06-05 02:06:40 -0400, Daniel Kahn Gillmor wrote:
> For reproducibility alone, the two relevant changes to apply if
> libisoburn debian packaging is rebuilding the docs from the texinfo
> would just be:
>
>
> http://libburnia-project.org/changeset/5190/libisoburn/trunk/xorriso/parse_exec.c
>
> http://libburnia-project.org/changeset/5190/libisoburn/trunk/xorriso/xorriso.texi
The attached patch works to just enable the c argument to -alter_date in
1.3.2-1.1. with that patch, i can reliably create reproducible ISOs by
sorting the extents and fixing the timestamps.
It looks like the libisoburn debian packaging isn't rebuilding the
documentation from the .texi sources, so the manpage and info pages
aren't getting updated with the new flags to -alter_date, though.
I'd be happy to NMU this targeted patch as an interim step toward
reproducibility unless the pkg-libburnia maintainers plan to upload the
new upstream release sometime soon.
George, maybe you can comment on those plans?
Regards,
--dkg
From a1af7514e72a7ba4ca788cc5b0f443a675e7b439 Mon Sep 17 00:00:00 2001
From: scdbackup
Date: Wed, 27 Nov 2013 09:51:05 +
Subject: [PATCH] New -alter_date types: a-c , m-c , b-c , c
git-svn-id: http://svn.libburnia-project.org/libisoburn/trunk@5190 fff8544e-141b-0410-a922-c1ea1f00b384
---
xorriso/parse_exec.c| 9
xorriso/xorriso.texi| 30 +---
2 files changed, 114 insertions(+), 65 deletions(-)
Minimized by Daniel Kahn Gillmor
--- a/xorriso/parse_exec.c
+++ b/xorriso/parse_exec.c
@@ -2654,12 +2654,21 @@
{
int ret;
+ *t_type= 0;
if(strcmp(time_type, "a")==0)
(*t_type)|= 1;
+ else if(strcmp(time_type, "a-c")==0)
+ (*t_type)|= 1 | 256;
else if(strcmp(time_type, "m")==0)
(*t_type)|= 4;
+ else if(strcmp(time_type, "m-c")==0)
+ (*t_type)|= 4 | 256;
else if(strcmp(time_type, "b")==0)
(*t_type)|= 5;
+ else if(strcmp(time_type, "b-c")==0)
+ (*t_type)|= 5 | 256;
+ else if(strcmp(time_type, "c")==0)
+ (*t_type)|= 2 | 256;
else {
sprintf(xorriso->info_text, "%s: Unrecognized type '%s'", cmd, time_type);
if(!(flag & 1))
--- a/xorriso/xorriso.texi
+++ b/xorriso/xorriso.texi
@@ -1946,9 +1946,18 @@
@item -alter_date type timestring iso_rr_path [***]
@kindex -alter_date sets timestamps in ISO image
@cindex Timestamps, set in ISO image, -alter_date
-Alter the date entries of a file in the ISO image. type is
-one of "a", "m", "b" for access time, modification time,
-both times.
+Alter the date entries of files in the ISO image. type may be one of
+the following:
+@*
+"a" sets access time, updates ctime.
+@*
+"m" sets modification time, updates ctime.
+@*
+"b" sets access time and modification time, updates ctime.
+@*
+"a-c", "m-c", and "b-c" set the times without updating ctime.
+@*
+"c" sets the ctime.
@*
timestring may be in the following formats
(see also section EXAMPLES):
@@ -1989,6 +1998,10 @@
conversion. E.g. 2013010720574700, 2013010720574700LOC.
The last two digits cc (centiseconds) will be ignored, but must be present
in order to make the format recognizable.
+@*
+Example:
+@*
+ -alter_date m-c 2013.11.27.103951 /file1 /file2 --
@c man .TP
@item -alter_date_r type timestring iso_rr_path [***]
@kindex -alter_date_r sets timestamps in ISO image
@@ -5883,8 +5896,11 @@
@node ExCharset, ExPseudo, ExBootable, Examples
@section Change existing file name tree from ISO-8859-1 to UTF-8
This example assumes that the existing ISO image was written with character
-set ISO-8859-1 but that the readers expected UTF-8. Now a new session with
-the same files gets added with converted file names.
+set ISO-8859-1 but that the readers expected UTF-8. Now a new session
+gets added with converted file names.
+Command -changes_pending "yes" enables writing despite the lack of any
+manipulation command.
+@*
In order to avoid any weaknesses of the local character set, this command
pretends that it uses already the final target set UTF-8.
Therefore strange file names may appear in messages, which
@@ -5895,7 +5911,7 @@
@*
-out_charset UTF-8 -backslash_codes on -dev /dev/sr0 \
@*
- -alter_date m +0 / @minus{}@minus{} -commit -eject all
+ -changes_pending yes -commit -eject all
@c man .SS
@c man .B Operate on storage facilities other than optical drives
@node ExPseudo, ExCdrecord, ExCharset, Examples
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#787795: grub: please build rescue ISO and floppy reproducibly
Source: grub Version: 2.02~beta2-23 Severity: wishlist User: reproducible-builds@lists.alioth.debian.org Usertags: timestamps Control: block -1 with 787793 The ISOs (/usr/lib/grub-rescue/grub-rescue-cdrom.iso and /usr/lib/grub-rescue/grub-rescue-floppy.iso) that are created by the grub build process embed subtle variations in the timestamps and the extents of each file. This is one of the things that keeps the package from producing byte-for-byte identical binary builds. (see https://reproducible.debian.net/rb-pkg/unstable/amd64/grub2.html) I think this can be fixed with a couple steps in debian/rules: sort the files by extent, and fix the timestamps. See some of the scripts sketched here for example: https://lists.gnu.org/archive/html/bug-xorriso/2015-06/msg00013.html However, it won't be completely reproducible until we get a newer version of xorriso in debian so that we can "-alter_date_r c" (see #787793, which blocks this bug). Note that there are other parts of the package which are also unreproducible (/usr/share/qemu/grub.bin and usr/lib/grub-xen/grub-i386-xen.bin binaries differ in ways i have not examined, and there are timestamps in usr/share/info/grub-dev.info.gz). These probably will be covered with separate bug reports. Regards, --dkg -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (500, 'testing'), (200, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#787793: libisoburn: new upstream version 1.4.0 available
Source: libisoburn Version: 1.3.2-1.1 Severity: wishlist User: reproducible-builds@lists.alioth.debian.org Usertags: timestamps toolchain libisoburn 1.4.0 is available upstream as of 2015-05-17. This update in particular includes the following changeset that allows for setting of ctime in the generated ISO, which is needed for being able to create reproducible ISOs with xorriso: http://libburnia-project.org/changeset/5190 For reproducibility alone, the two relevant changes to apply if libisoburn debian packaging is rebuilding the docs from the texinfo would just be: http://libburnia-project.org/changeset/5190/libisoburn/trunk/xorriso/parse_exec.c http://libburnia-project.org/changeset/5190/libisoburn/trunk/xorriso/xorriso.texi But upgrading to the newest version would avoid the need for any sort of special patches. Thanks for maintaining libisoburn in debian! Regards, --dkg -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (500, 'testing'), (200, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#787444: help2man: support externally-supplied --date for reproducibility
On Thu 2015-06-04 21:51:37 -0400, Brendan O'Dea wrote: > Local times, and daylight savings are just too much of a PITA. Just > use UTC and if builds on the first of the month are possibly different > to the changelog, so be it. I agree with Brendan here. If there are no objections to the idea that we're sticking with ISO-8601 in UTC, preferably with the Z suffix (e.g. 2015-06-05T01:08:20Z) then we just need to settle on what the right name is. Lunar^, you'd mentioned that there had been discussions about a preferred variable name that you might like better than SOURCEDATE. Any preference? Where should we document this? --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] generating reproducible ISOs with xorriso
On Thu 2015-06-04 14:08:36 -0400, Thomas Schmitt wrote: > The syntax would have to be different and probably a more > comprehensive name will come to us when we know what xorriso > features in particular shall be bundled with the new command. That seems like a reasonable approach. > The users will have their own problems to get the new concept > integrated into their ISO production. > > The main users within Debian are debian-cd and Debian Live. > One should develop proposals how to make their ISOs reproducible > and how to publish the parameters (e.g. the time stamp) which > they will have to provide to xorriso when reproducing the ISOs. > > A special user is grub-mkrescue. We will have to come up with > a well motivated use case to convince Vladimir Serbinenko. That's fine; we can also patch grub within debian to do the right thing until we've gotten Vladimir properly convinced :) I've already convinced Vladimir that he should use a stable embedding timestamp when creating PE executables, so he's aware of the general concern about reproducibility: http://git.savannah.gnu.org/cgit/grub.git/commit/?id=85a7be2414c4718e96d81a2ebaa70d0d42152e62 I can imagine that he would also like to be able to say that "grub-mkrescue", executed twice with the same arguments on different machines (running the same versions of grub) should be able to produce the same .iso image. I'll think about how to pose that to the grub mailing list once we've sorted out how to do it. >> https://reproducible.debian.net/dbd/unstable/amd64/grub2_2.02~beta2-23.debbindiff.html >> if i'm reading that right, then in >> /usr/lib/grub-rescue/grub-rescue-cdrom.iso, we have >> /BOOT/GRUB/GRUB.CFG;1 with extent 2316 in the first build, and extent 47 >> in the second. > > Strange ... i had a different model in mind. > Will have to review the source of libisofs (ecma119.c, i guess). > > How were the two compared ISOs generated ? > By the same xorriso version and same options resp. commands ? yes, the same set of packages (the "build-deps", in debian), running with the same source code (the grub source, which invokes xorriso). You can see the full build output (warning! > 30MiB!) here: https://reproducible.debian.net/rbuild/unstable/amd64/grub2_2.02~beta2-23.rbuild.log It looks like it's being called from within grub-mkrescue, once for the floppy and once for the CD image. This comes from https://sources.debian.net/src/grub2/2.02~beta2-22/debian/rules/#L242 -- pkgdatadir=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc \ obj/grub-pc/grub-mkrescue \ --directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core \ --locale-directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po \ --output=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso \ /tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk obj/grub-pc/grub-mkrescue: warning: cannot open directory `/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po': No such file or directory. xorriso 1.3.2 : RockRidge filesystem manipulator, libburnia project. Drive current: -outdev 'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso' Media current: stdio file, overwriteable Media status : is blank Media summary: 0 sessions, 0 data blocks, 0 data, 83.4g free Added to ISO image: directory '/'='/tmp/grub.MBEdSo' xorriso : UPDATE : 277 files added in 1 seconds Added to ISO image: directory '/'='/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk' xorriso : UPDATE : 280 files added in 1 seconds xorriso : NOTE : Copying to System Area: 512 bytes from file '/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/boot_hybrid.img' ISO image produced: 2467 sectors Written to medium : 2467 sectors at LBA 0 Writing to 'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso' completed successfully. # save space for floppy image rm -rf obj/grub-rescue-pc/grub-core/po obj/grub-rescue-pc/unicode.pf2 pkgdatadir=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc \ obj/grub-pc/grub-mkrescue \ --directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core \ --locale-directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po \ --output=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-floppy.img \ --compress=xz \ -- -no-pad /tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk obj/grub-pc/grub-mkrescue: warning: cannot open directory `/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po': No such file or directory. xorriso 1.3.2 : RockRidge filesystem manipulator, libburnia project. Drive current: -outdev 'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-floppy.img' Media current: stdio file, overwriteable Media status : is blank Media summary: 0 sessions, 0 data blocks, 0 data, 83.4g free Added to ISO image
Re: [Reproducible-builds] Bug#787444: help2man: support externally-supplied --date for reproducibility
On Thu 2015-06-04 10:38:53 -0400, Ximin Luo wrote: > A few months ago I had an idea for this that would be more generalisable. See > > https://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20150330/001317.html > > for details. TL;DR is to have SOURCEDATE as an environment variable in > ISO8601 format. I'm fine with this part of your proposal. Your proposal also includes a bunch of workarounds with faketime, which i'm a little concerned about (i say this as the faketime maintainer in debian, not wanting its particular flakiness in the build path for everything). Can we separate the $SOURCEDATE part of your proposal from the faketime part and just work on $SOURCEDATE independently? What TZ should SOURCEDATE have? ISO8601 is capable of supplying a TZ as well, so the current time could be written in a wide variety of ways while meaning the same instant: 0 dkg@alice:~$ date '+%FT%T%z' && date -u '+%FT%T%z' 2015-06-04T13:25:25-0400 2015-06-04T17:25:25+ 0 dkg@alice:~$ I feel like we should we always set it to UTC, so that the inbound parsed offset would be +. sound sensible? > It would be awesome for help2man to support this. help2man (and any other tool that accepts $SOURCEDATE) would also need to ensure that it extracts the parts it wants in a TZ-independent fashion as well. (not parsing back to localtime) > At some point, debhelper can even set this environment variable > automatically. We should open a bug with debhelper requesting this feature as soon as we come to agreement on the name and semantics. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] generating reproducible ISOs with xorriso
On Thu 2015-06-04 09:53:11 -0400, Thomas Schmitt wrote: > I saw some sin-list page about packages shortly after > xorriso-1.4.0 was released. > It complained about libburn's doc/doxygen.conf.in which i hope to > have fixed by > http://libburnia-project.org/changeset/5446 Thanks! We don't think it's a sin, but you're absolved anyway :) > Afaik, GRUB2 uses the ISO 9660 Modification Date as UUID of the ISO. > There are several ISO 9660 timestamps. One would have to fixate > them by explicit xorriso commands. We can certainly give patches to grub to do that explicitly (and we might come ask questions here if we can't figure out the right commands). If we can identify the specific commands (beyond what you point out below), would there a general interest upstream in something like a --reproducible=TARGETDATE, which would make these decisions explicitly within xorriso, rather than every invocation having to learn the full suite of variations? It's probably easier to ask grub upstream (and other xorriso users) to add a single flag than to add a complex set. > The dull ISO 9660 names get sorted and their data content > is stored in the sequence of that sorting. So if the tree of > paths stays unaltered, the sequence of data extents should > stay unaltered too. > The mapping from Rock Ridge names to ISO 9660 names is supposed > to be deterministic in this case. (It is complicated, though.) Hm, this *doesn't* seem to be what's happening with the grub ISO: https://reproducible.debian.net/dbd/unstable/amd64/grub2_2.02~beta2-23.debbindiff.html for example, if i'm reading that right, then in /usr/lib/grub-rescue/grub-rescue-cdrom.iso, we have /BOOT/GRUB/GRUB.CFG;1 with extent 2316 in the first build, and extent 47 in the second. Am i reading that right? I think the contents of the files in the ISO should already be stable across builds, and it's just the extents that are changing. Note that the build-host filesystems varied, and the generated files may have been placed in the build-host filesystems at different times on each build. Maybe it's still sorting by dirent order instead of by ISO9660 names? > The second number after "Bootoff" is the block address of the > data extent of an El torito boot image data file. Thus covered > by the sorting if all file sizes are unaltered. OK, that's good -- we've seen this pattern before, where fixing some known variations resolves some more complex variations (GNU buildid is a classic example). I'm happy to ignore that until we get the extent business sorted, and then maybe it will go way on its own :) > I riddle what the first number might be. > debian-7.7.0-amd64-netinst.iso has "Bootoff 350 848". > The first El Torito boot image begins at block 848 decimal. > Maybe just a disguised hex ? 0x350 = 848 yes, i think that's correct. You can also see those figures in hex in /boot.catalog > Well possible. > After the usual xorriso -as mkisofs options of debian-cd > one could add native xorriso commands. Use a timestamp > format that is not prone to time zone computation. E.g. > the native format of ISO 9660: > > timestamp="2015060415363300" > xorriso -as mkisofs ...all.the.debian-cd.options... \ > -- \ > -volume_date "c" "$timestamp" \ > -volume_date "m" "$timestamp" \ > -volume_date "x" "default" \ > -volume_date "f" "default" \ > -alter_date "b" "$timestamp" / -- \ > -alter_date "c" "$timestamp" / -- \ thanks, this is very useful! > Filesystem times "x" = Expire and "f" = Valid Since are best set > to the default value 000...000. Just in case anybody cares to > obey them. i had no idea that these flags even existed, scary... > One will have to make experiments with the ISO production > scripts which shall be used. Identify the deviating spots > in the ISO, their meaning, and their origin. > > I am willing to help with advise and/or necessary new features. Thanks for your help already, Thomas, and for your willingness to help in the future. Any thoughts about the extent issue described above? Regards, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#787444: help2man: support externally-supplied --date for reproducibility
On Thu 2015-06-04 08:17:59 -0400, Brendan O'Dea wrote:
> The idea was that unless that environment variable was set, the
> behaviour would be unchanged: the current date would be used. It
> seems unlikely that DEB_BUILD_CHANGELOG would be set accidentally.
right, but this wouldn't be terribly useful for folks like fedora, who
also ship help2man and might want to be reproducible:
https://admin.fedoraproject.org/pkgdb/package/help2man/
http://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/
>> If the build system is going to set environment variables, maybe it
>> could set an environment variable with a parseable date string
>> (ISO-8601?) and help2man could look at that environment variable and
>> extract a string from it? (ideally this extraction step would be
>> TZ-independent, too)
>
> That would also work, and if your reproducible build environment was
> to pull the changelog date and stick it into an environment variable,
> then you should only need to patch the tools rather than the Makefile
> of every package which uses them.
>
> At worst, for the tools you can't patch, you're in the same place you
> are now--having to patch all the Makefiles, and even that is perhaps
> easier as you already have the date:
>
> --date=${DEB_BUILD_CHANGELOG_DATE:-$(date --iso-8601=seconds)}
Yep, i think this makes sense; reproducible-build folks -- do we want to
take this tack with help2man? If so, what would the environment
variable be named? what would its contents be?
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] generating reproducible ISOs with xorriso
Hi libburnia/xorriso folks-- I participate in the Debian Reproducible Builds project [0] (cc'ed here). Our goal is to ensure that free software can be built from source in a way that the binary outcome is byte-for-byte identical, so that compromised build infrastructure can be detected. One of the things that introduces variation in binaries are packages that build ISOs using xorriso. I wanted to see if xorriso would be interested in offering a "reproducible" option during ISO creation. The variation within an ISO can come from many places, probably including: * filesystem timestamps * extent ordering/numbering (maybe derived from source filesystem ordering) * bootable metadata (Boot offsets? i don't know the jargon, but there is a value reported by "isoinfo -d" called "Bootoff") One example of a package that has unreproducible ISOs is grub: https://reproducible.debian.net/rb-pkg/unstable/amd64/grub2.html We can try to minimize the external variations before building an ISO (e.g. by "touch"ing all the source files to a static timestamp, and maybe by sorting the files before generating a manifest to send to xorriso?), but it seems like it would be simpler if there were a way to tell xorriso to just make an identical image with all metadata standardized in some way. This mode might imply: * supplying a timestamp to be used for all imported files (like alter_date_r ?) * sorting files included so that extent numbering is constant * ... other things? I don't know enough about how xorriso works to know what else would be usefully standardized to make ISO creation byte-for-byte repeatable, but I figure you do :) Maybe this is actually already possible with xorriso, and i just need to do add a few simple switches? If so, do you have suggestions? Thanks for your work on libburnia! Regards, --dkg [0] https://wiki.debian.org/ReproducibleBuilds/ ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [PATCH] provide anchors to target sections of the front page
On Thu 2015-06-04 05:07:51 -0400, Holger Levsen wrote: > Cool! Thanks, merged + deployed your patch! great, thank you, Holger! > Will you be providing links to these anchors too? ;-) Yes, but they'll probably be external, like https://www.debian-administration.org/users/dkg/weblog/115 If folks want to cross-link internally too, that would be great, but i was just scratching my particular itch here :) --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#787444: help2man: support externally-supplied --date for reproducibility
Hi Brendan-- Thanks for the quick followup! (cc'ing the r-b list) On Wed 2015-06-03 08:24:51 -0400, Brendan O'Dea wrote: > On 2 June 2015 at 04:49, Daniel Kahn Gillmor wrote: >> Please consider the attached patch as a step on the way toward allowing >> packages that use help2man to build reproducibly. For more info, see: > > I appreciate that you've provided a relatively generic solution to > your problem: adding an option to set the date, but it occurs to me > that most users don't actually care. Additionally, this will require > changing the builds for anything which uses help2man to add a --date > option. You're right, and that might make the fix take longer to take effect. But it's not a terrible cost, and we have other packages in that same situation. > My inclination is instead to do something fairly specific to your > project: If the environment variable DEB_BUILD_CHANGELOG (or something > similar) is set, then the file to which it refers will be used to find > the latest revision date, and that date will be used on the manual > pages. Presumably the build system could set this prior to > build/test? i think this could work, but it might not apply particularly well to anyone outside of debian who uses help2man. I note that help2man is a native package -- do you know if it's used outside of debian at all? Making the help2man run dpkg-parsechangelog also seems a little hairier than it ought to be, and introduces a dependency on dpkg-dev which seems a little weird. If the build system is going to set environment variables, maybe it could set an environment variable with a parseable date string (ISO-8601?) and help2man could look at that environment variable and extract a string from it? (ideally this extraction step would be TZ-independent, too) What do you think? --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] python-support umask affects reproducibility of built packages
Control: user reproducible-builds@lists.alioth.debian.org Control: usertags 588746 + umask Control: affects 588746 xtalk Fixing the python-support umask issue for .private files would make more packages build reproducibly. For example, consider the xtalk package, which builds reproducibly *except* for the umask difference in usr/share/python-support/xtalk.private, which appears to be generated by python-support: https://reproducible.debian.net/rb-pkg/unstable/amd64/xtalk.html --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] [PATCH] Bug #777753: GCC variation with link-time optimization (LTO)
When GCC does link-time optimizations, it embeds some randomness in the generated object. This is https://bugs.debian.org/53, and it affects several packages. it may be the underlying cause of several instances of build_id_variation_requiring_further_investigation, but i haven't removed those issues yet, because there may be other build_id variation mechanisms on the packages affected by LTO as well, and i haven't tested them all. The changes to packages.yml were found based on the "affects:" annotations on #53. --- issues.yml | 8 packages.yml | 47 +++ 2 files changed, 55 insertions(+) diff --git a/issues.yml b/issues.yml index 1fb1c6d..c83a886 100644 --- a/issues.yml +++ b/issues.yml @@ -505,3 +505,11 @@ timezone_variance_because_of_automake_mdate: automake's mdate-sh script pretty-prints the modification time of a file. This output can influence further steps in the build process and make it unreproducible. url: https://wiki.debian.org/ReproducibleBuilds/TimezoneVarianceBecauseOfAutomakeMdate +randomness_in_lto_debug_symbols: + description: | +GCC-produced binaries that use link time optimization (LTO) embed +randomized intermediate filenames (e.g. cc.ltrans.[0-9]+) +in generated debug symbols. +. +see also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65015 + url: https://bugs.debian.org/53 diff --git a/packages.yml b/packages.yml index 4b63e19..ca8afc8 100644 --- a/packages.yml +++ b/packages.yml @@ -381,6 +381,10 @@ apt: - timestamps_from_cpp_macros bugs: - 774342 +apt-cacher-ng: + version: 0.8.0-3 + issues: +- randomness_in_lto_debug_symbols apt-dater: version: 1.0.1-1 issues: @@ -601,6 +605,10 @@ avfs: version: 1.0.1-2 issues: - bad_handling_of_extra_warnings +avian: + version: 1.1.0-4 + issues: +- randomness_in_lto_debug_symbols avinfo: version: 1.0.a15+20090102-1 bugs: @@ -745,6 +753,7 @@ batmand: version: 0.3.2-15 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols bb: version: 1.3rc1-8.1 issues: @@ -1549,6 +1558,7 @@ cloop: issues: - timestamps_in_tarball - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols closure-compiler: version: 20130227+dfsg1-8 issues: @@ -2937,6 +2947,7 @@ encfs: version: 1.7.4-5 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols enemies-of-carlotta: version: 1.2.6-4 issues: @@ -3200,6 +3211,10 @@ evolution-webcal: version: 2.32.0-2 bugs: - 759823 +exactimage: + version: 0.8.9-7 + issues: +- randomness_in_lto_debug_symbols exifprobe: version: 2.0.1-3 comments: | @@ -6060,6 +6075,10 @@ kvpnc: version: 0.9.6a-2.1 issues: - build_id_variation_requiring_further_investigation +kwalletcli: + version: 2.12-3 + issues: +- randomness_in_lto_debug_symbols kwin-style-crystal: version: 2.2.1-2 issues: @@ -10367,6 +10386,10 @@ maitreya: version: 7.0.7-1 issues: - timestamps_from_cpp_macros +makefs: + version: 20100306-5 + issues: +- randomness_in_lto_debug_symbols makehuman: version: 1.0.0~alpha6-5 issues: @@ -11065,34 +11088,43 @@ mupen64plus-audio-sdl: version: 2.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-core: version: 2.0-7 comments: | Fails to build reproducibly due to link-time optimization (-flto) + issues: +- randomness_in_lto_debug_symbols mupen64plus-input-sdl: version: 2.0-4 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-rsp-hle: version: 2.0-3 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-rsp-z64: version: 2.0.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-ui-console: version: 2.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-video-arachnoid: version: 2.0.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-video-glide64: version: 2.0.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-video-glide64mk2: version: 2.0-3 issues: @@ -11101,10 +11133,12 @@ mupen64plus-video-rice: version: 2.0-3 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols mupen64plus-video-z64: version: 2.0.0-2 issues: - build_id_variation_requiring_further_investigation +- randomness_in_lto_debug_symbols muse-el: version: 3.20+dfsg-0.1 issue
Re: [Reproducible-builds] sbuild: predictible build location for reproducible builds
On Fri 2015-04-03 06:46:07 -0400, Holger Levsen wrote: > I like /usr/src/debian/$packagename-$random where $random are 4 or 8 > alphanumberic characters, as there might be situations where one wants to > build the same package (+version+suite) simultaneously on the same host. > > Thinking further, /usr/src/debian/$packagename_$version_$random might be > better. Holger's proposal here seems to be in direct opposition to the stated goal of #778571, which is to have the directory and path be precisely predictable. How do we get these goals to align? --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Variation in day
On Tue 2015-02-24 16:48:44 -0500, Jérémy Bobbio wrote: > Reiner Herrmann: >> after Faux found a package with a variation not in time, but only >> in the date [1] because it was built around midnight, I had an idea >> about how to get a variation in the day, without having to change >> the clock of the host. >> >> Varying the timezone between the first and second build to the >> lowest and highest possible offset gives a difference of about >> one day: >> >> $ TZ=/usr/share/zoneinfo/Etc/GMT+12 date >> Tue Feb 24 09:05:40 GMT+12 2015 >> $ TZ=/usr/share/zoneinfo/Etc/GMT-14 date >> Wed Feb 25 11:05:49 GMT-14 2015 >> >> It is even possible to cover a day and 2 hours. :) >> (Perhaps it's even possible to create custom timezones, that >> would allow even larger variations like a year or so...?) >> >> What do you think about this approach? >> Would there be any disadvantages, and does anyone know >> of tools that ignore the TZ setting? > > That would be the difference between gmtime(3) and localtime(3). So this > will not catch tools using time in UTC format. > > But timezone is a worthwhile variation in itself! :) Agreed. This should be part of the varied environment during a rebuild, even if it doesn't catch tools that use the UTC format. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Doxygen issue with reproducible builds
On Sun 2015-02-22 03:37:41 -0500, marivalen wrote: > On 02/16/2015 02:43 PM, marivalen wrote: >> Furthermore the Doxyfile configuration file embedded in each package >> using doxygen is often copied from the upstream defaults and thus >> enables HTML_TIMESTAMP. The following query on codesearch.d.n shows that >> 277 source packages are affected: >> http://codesearch.debian.net/results/HTML_TIMESTAMP%20%2B%3D%20YES%20path%3ADoxyfile/page_0 >> >> >> So no matter whether my patch to doxygen is applied or not, these 277 >> source packages will either need individual fixes or a change to doxygen >> which allows to ignore the setting of HTML_TIMESTAMP in Doxyfile. I can >> implement this but this again has to be discussed with the Debian >> maintainers and doxygen upstream. > > As explained in my last email any reasonable fix (the unreasonable being > letting doxygen in Debian never output timestamps) requires patching the > 200+ source packages affected by the issue individually. > > Thus, if there is no objection I will proceed with filing bugs with > patches against the affected source packages and let them be blocked by > bug #778490 if the source package uses Latex. This sounds reasonable to me. Thank you for doing this, --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [Mono-dev] making mono builds reproducible (xamarin bz #26842)
On Mon 2015-02-16 18:17:53 -0500, Michael McGlothlin wrote: > I'd always store time in epochs. Seconds since 1/1/1970 GMT. > > The use of textual date strings instead of a epochs is one of the > worst things I've seen from the C# way of doing things. I had often > wondered why so many programs could have so much trouble with handling > dates and times correctly.. I agree that silly standards like RFC 822 timestamps are crazy and should not be used anywhere we can avoid them. However, the ISO-8601 date/timestamp format is both human- and machine-parseable, whereas most humans can't look at a UNIX epoch timestamp and know even whether it's in the past or the future. That said, i really care more about reproducibility than i do about any particular timestamp format. if folks are fine with UNIX epoch timestamps and with the environment variable interfacfe Jo proposes, i'll be happy with that. Is this something that could be adopted upstream? --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] making mono builds reproducible (xamarin bz #26842)
Hi Mono folks--
some good discussion has come up on the xamarin bugtracker about being
able to make builds using the mono toolchain reproducible:
https://bugzilla.xamarin.com/show_bug.cgi?id=26842
Jo Shields offered a one-liner fix to PEWriter.cs to allow the use of an
environment variable to fix the timestamp epoch, but made it clear that
more feedback is needed, so i'm raising it here:
from:
public DWORD TimeDateStamp = (uint)(DateTime.UtcNow - new DateTime(1970, 1,
1, 0, 0, 0, DateTimeKind.Utc)).TotalSeconds;
to:
public DWORD TimeDateStamp =
Environment.GetEnvironmentVariable("IKVM_WRITER_TIMESTAMP_EPOCH") != null ?
uint.Parse(Environment.GetEnvironmentVariable("IKVM_WRITER_TIMESTAMP_EPOCH")) :
(uint)(DateTime.UtcNow - new DateTime(1970, 1,
1, 0, 0, 0, DateTimeKind.Utc)).TotalSeconds;
(i'm not sure whether the integer number of seconds is the best form for
the environment variable, or whether it would be better to parse a
standard date string -- from debian's perspective, we can deal with
either, of course)
And this still doesn't solve the secondary issue of the assembly GUID,
as Jo noted.
Any thoughts about how to best enable binary-reproducible builds from
the mono toolchain if the invoker requests them?
Regards,
--dkg
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
On Mon 2015-02-16 04:39:32 -0500, Hans-Christoph Steiner wrote: > I think this topic is far too vast with far too many dependencies to really > have a useful discussion on without a full time, dedicated team. Since that > seems highly unlikely in the near future, we need to break it down into chunks > of work that we can achieve with the time and resources we actually have. > > So we need to focus on drilling down to what is the simplest useful form of > package signing that will cause the least amount of problems when we decide to > change how package signing works. This means we get a prototype out as soon > as possible, and we can learn a lot from that. I think that's pretty easy to > do, something like this: > > * make dpkg optionally check package sigs, and refuse to install on bad sig > * use apt signing model: signatures verified from the apt key ring > * signing can start happening in the build tools, by the uploader > * start work towards getting the Debian built/apt infrastructure signing The above bullet points don't seem mutually compatible. if the uploader is doing the per-package signing, then their key won't be in the apt keyring; this would make each package have a bad sig; and would mean that dpkg in signature-checking mode would reject the packages. Choosing the apt keyring as the signing model appears to punt entirely on questions of corroborative authority and how that plays out in a multi-distro ecosystem. But overall, I think this discussion of embedded per-package signatures misses the point: the only reason we're talking about possible future per-package signatures is because we want to avoid breaking them as we add mechanisms to support reproducible builds. We *have* a team actively working on reproducible builds (this mailing list), who have done a lot of work already and have thought about the needs for that project. If we don't have a use case or a plan or a team or a goal or known constraints for doing embedded per-package signing, then delaying the archive and system changes needed by reproducible builds on behalf of some non-concrete desire for per-package signatures is a mistake. We shouldn't be blocking progress on an active and effective project for one that isn't underway at all. And remember that we already have a very clear understanding of how to do *non-embedded* per-package signatures, should anyone want to do them. My pkg-fingerprint proposal was trying to lay out compromise that would leave per-package embedded signatures possible in the future without having to specify them in detail today. I don't *want* to specify them in detail today. I want to be able to move on with making the archive reproducible :) --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Reproducible Builds — proof of concept successful for 83% of all sources in main
On Fri 2015-02-13 14:33:12 -0500, Paul Gevers wrote: > On 13-02-15 18:28, Reproducible builds folks wrote: >> If you want to help, a first step is to check the reproducibility of >> your packages [DDLIST]. Feel free to ask for help on the >> mailing list or in >> #debian-reproducible on irc.debian.org. > > It would help me if you would have mentioned here what you expect people > to do when their package is not reproducible, but when this is the > result of other packages that they use during building. I am not going > to fix my packages if e.g. the timestamp in the documentation is > inserted by the use of formatXtoFormatY, we should then rather focus on > fixing formatXtoFormatY. I believe you agree. If people all start fixing > such issues inside their own package than in some time we have all kind > of solutions, which may (or may not) bit-rot and may not be needed if we > fix formatXtoFormatY. Some of these tools have legitimate reasons in non-reproducible environments to inject timestamps. Even in reproducible environments, it can be useful to inject a timestamp (a known, static one like the value in debian/changelog) in things like generated man pages. This means we should encourage formatXtoFormatY tools to make it easy to expose no-timestamp and/or externally-provided-timestamp modes of operation, and we should certainly encourage package-building frameworks like dh to set those modes by default. The less information we have to encode (and maintain) in every single debian/rules or debian/control file, the better. However, for packages that don't use a framework we can fix, or which use a tool that has no plans to adopt these kinds of modes upstream, having the deviations embedded in our packaging files is the only thing to do if we want reproducible builds. It would be a shame if we had to wait for every format-converter to get upgraded to do the right thing automatically to get debian itself to a reproducible state. Your concerns about bit-rot are legitimate, though; if a piece of the toolchain is where a "proper" fix belongs, and all the dependent packages are working around it now, perhaps we could (a) make sure that the "proper fix" bug report is filed in the bts against the piece of the toolchain, and that it is marked as "affects:" with all the packages that are currently working around it. If a package is doing a workaround that it shouldn't need to do, we could even file a wishlist bug against that package and mark it as blocked by the toolchain bug. Basically, i think the BTS has the semantics to support keeping track of these dependent issues so that we end up with clean long-term fixes, while we can use our packager's discretion to implement the shorter-term workarounds, annoying though they may be. what do you think? --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [Reproducible-commits] [notes] 01/01: add new issue, randomness_in_gnu_build_id and, so far, two packages affected by it: encfs and bacula
On Fri 2015-02-13 11:10:31 -0500, Holger Levsen wrote: > Hi, > > On Samstag, 7. Februar 2015, Jérémy Bobbio wrote: >> I think it really is not helpful. It's like having a category >> “needs_more_work_to_understand_the_problem”. > > actually I think such a category, or even such categories, would be helpful. > > I wouldnt want to work on this though if I'm the only person who thinks so. based on https://reproducible.debian.net/index_issues.html, 18027 reproducible packages out of 21579, meaning 3552 packages are unreproducible. the page also says "a total of 2070 packages categorized in 72 issues" (i think this is wrong, btw; i think there are 2070 categorizations, but some packages have more than one categorization (e.g. debian-policy has for categorizations), so the total number of packages is probably less, but i'll go with it for now) That means that 1482 packages are not categorized at all. (another side note: can we get a view on the website of packages that *have not* been categorized?) I think having a category of "investigated and needs more work" would be useful. For newbies looking to help, they could avoid those packages. And for more experienced people, they could treat that tag as an interesting challenge. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
On Fri 2015-02-13 03:36:20 -0500, Hans-Christoph Steiner wrote: > I think it would be much simpler to just have the single package signature > that is embedded in the package file itself, like Android APKs and Java JARs. > Since the package is built reproducibly, anyone who builds it can just copy > the canonical signature into their copy of the package they just built, and > it'll match the sha512sum of the signed apt metadata. It seems like you're saying everyone will be able to agree on which signing authority is "canonical" for any given package. I'm not convinced that's the case. > The big question there is determining the where the canonical > signature happens. It seems like it should be the official Debian > build process, since it is the only process guaranteed to be the same Even though i'm personally likely to treat Debian as "the canonical source" i care about, i don't want it to be that way. I would like Debian to be able to be a downstream as well as an upstream (see the work feeding back into debian from ubuntu, for example); if a .deb package can contain an internal signature, and i'm looking at a given .deb in isolation on my debian system, i want to see it signed *by debian*, not by whoever happened to produce it first. Otherwise, it's not clear to me that the embedded signature is useful to me as an end user at all. > Another question is whether dpkg checks whether the signers match when > upgrading, like the Android model (a package can only be upgraded by > another package signed by the same key). This would be nice, but > seems optional and hard to do in Debian. Maybe this is the question we need to answer to move the discussion forward to make sure we're taking the desire for embedded signatures into account when thinking about reproducible .debs: how exactly do we expect an embedded signature to be used/evaluated? by who, and in what context? > I think the .buildinfo file is useful, but for a separate process. It should > be the canonical file for running a reproducible build. I'm not sure what this means. I'd be very happy if *all* of my debian packages were reproducible builds, and i could have a way of verifying it. I'd consider that more valuable than knowing that all my .debs were signed by any individual authority. So if we're really talking about a tradeoff between signed buildinfo files and signed packages, i'd certainly prefer signed buildinfo files. But my proposal was an attempt to let people have both, without forcing the entire ecosystem to agree on "who is a canonical authority for package X, without whom a reproducible package is impossible" --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
On Fri 2015-02-06 01:13:18 -0500, Guillem Jover wrote:
> Take the example I gave previously of a binary package detached from
> an archive, just a .deb package laying around, either from an old
> download or passed to you by someone. You have to *know* the origin of
> the binary, otherwise you need to first start hunting down where this
> binary was built, say Debian, one of its derivatives or even somewhere
> else. And sure, once that's known, the user *might* possibly be able to
> reproduce the build, but I don't see many (if not most) users being able
> or willing to set up a reproducible build environment just to verify
> where a binary was coming from (say my relatives). If you cannot or
> wont do that, you need to trust the distribution, the remote server,
> the network, the remote binary including any possible reproducible
> information being correct. At that point you or a program might as well
> have just verified an embedded signature.
I see per-package signatures and reproducible builds as complementary to
one another. I'd be happy to brainstorm ways that we can make sure we
can get the advantages of both.
I understand the goals of per-package signatures, and that it's more
convenient for most folks to have a single file than to have to schlep
around multiple files to be able to verify the package.
I also understand the ecological benefits of reproducibility, especially
for a project like debian that has so many heterogenous downstreams that
pull (directly or indirectly) from it.
For one of the big reproducibility benefits -- detecting compromised
build environments -- to accrue to everyone, we only need a handful of
independent reproducers who can't all be compromised together, and a
straightforward way that people can compare the results and raise an
alarm if something looks fishy.
The benefits of reproducibility come from *corroboration*. otoh, the
usual view of embedded package signatures come from *single authority*
("the origin of the binary"). For the broader ecosystem of debian + its
derivatives, where some binary packages might be shipped by multiple
vendors, it is actually useful to users of a derivative that doesn't
rebuild everything to be able to say with confidence "this package is
exactly the same as the packages debian ships". At the same time, as a
user of the derivative, you'd also like to be able to know that any
given package is "endorsed" or included by your distro, so that
arbitrary packages from debian proper can't just be slipped by your
package manager without knowing about it.
So there's this interplay (tension?) between authority of origin and
corroboration that i suspect we'd all like to support.
I've been partial to the separate .buildinfo file specifically because i
think i can see the path where we can get both corroboration and
specific authority from it (a single .buildinfo file can be signed by
multiple authorities), while the identical .deb in two distros can be
bitwise-compared using simple tools.
That said, this clearly doesn't address the convenience/portability
win of embedded package signatures raised by Guillem.
As a step toward brainstorming, here's a way that we could (maybe?) get
both:
* we could make a pkg-fingerprint tool that takes a package and
produces a cryptographically-strong fingerprint of the contents of
the package *minus* the signature elements.
As long as the package format has no internal signature, this would
just be something like sha512sum over the entire package. When we
have a package with an embedded-signature format, we'd need to define
a way to strip any/all signatures from the package in a reproducible
way that does not touch the rest of the contents of the package.
then pkg-fingerprint becomes something like:
pkg-strip-sigs < foo.deb | sha512sum
and the comparison between packages moves from /usr/bin/cmp to
comparing the outputs of pkg-fingerprint (or we could make a pkg-cmp
tool).
the buildinfo files would then store the pkg-fingerprint output over
the .deb or .rpm binaries that they produced.
This is a little bit inelegant, but maybe it's a way that we can have
our cake and eat it too? I'd love to hear other suggestions.
--dkg
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] don't panic? or: inform d-d-a
On Mon 2015-02-09 21:01:46 -0500, Jérémy Bobbio wrote: > Mattia Rizzolo: >> I added a lot of stuff to the pad. Not sure if everything is relevant, >> but yet, imho is nice to have. > > And I've trimmed a lot of it. ;) > > > I'm quite happy with the content as of 2015-02-10 02:00 UTC. I would be > in favor of sending the message in 24 or 48 hours to leave a bit more > time for extra reviews. It looks quite good to me right now. I added a Mail-Followup-To: header that encodes the recommendations in the "Further Discussion" line so that reasonable MUAs will do the Right Thing. We should send it out soon! --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] don't panic? or: inform d-d-a
On Sun 2015-02-08 06:58:37 -0500, Mattia Rizzolo wrote: > On Sun, Feb 8, 2015 at 12:48 PM, Holger Levsen wrote: >> The following is a suggestion which I'm not entirely happy with, feedback >> much >> welcome. I think we should probably move this to etherpad/gobby and >> fisnih/rewrite it collaborativly there. > > https://titanpad.com/AYVtDyCLDO > > :) > > please do edit it! I've given it a few minor edits. I think an announcement to d-d-a is a great idea: we need to make sure this work is on the radar of debian developers, and raising its profile like this is the right thing to do right now. One of the big things that i hope comes out of this project is a broad sense of best practices for software developers that normalizes the idea that reproducibility is a critical goal. It will probably take years for these ideas to really take root, but publicity like this is what needs to happen to get the ball rolling. --dkg ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#774422: perl: please make perl builds reproducible
On Fri 2015-01-23 06:03:25 -0500, Holger Levsen wrote:
> Hi Niko,
>
> On Freitag, 23. Januar 2015, Niko Tyni wrote:
>> A quick search indicates that there's no separate namespace for other
>> uname(2) information than the host name and domain name. This suggests
>> that something like http://www.bstern.org/libuname/ is needed. I'm not
>> aware of anything in Debian already that does that. Time for an RFP maybe
>> :)
>
> it builds fine but doesn't work:
>
> jenkins@jenkins:~/u/libuname-1.0.0$ make
> gcc -Wall -Werror -O2 -fPIC -c -o libuname.o libuname.c
> if [ "`uname -s`" = "SunOS" ]; then \
> ld -G -dy -z text -Qn -o libuname.so libuname.o; \
> else \
> ld -shared -fPIC -o libuname.so libuname.o; \
> fi
> jenkins@jenkins:~/u/libuname-1.0.0$ LD_PRELOAD=$PWD/libuname.so
> LIBUNAME='Linux;bar;2.6.15;#1;Mon Feb 37 22:33:44 UTC 2006;i686;unknown'
> uname
> -a
> uname: symbol lookup error: /var/lib/jenkins/u/libuname-1.0.0/libuname.so:
> undefined symbol: dlsym
This is resolved by the attached patch.
--dkg
diff --git a/Makefile b/Makefile
index 7c5fed4..4ecec59 100644
--- a/Makefile
+++ b/Makefile
@@ -4,7 +4,7 @@ CFLAGS=-Wall -Werror -O2 -fPIC
##CFLAGS=-Wall -Werror -O2 -fPIC -m64
CC=gcc
-LINUX_LDFLAGS=-shared -fPIC
+LINUX_LDFLAGS=-shared -fPIC -ldl
SOLARIS_LDFLAGS=-G -dy -z text -Qn
.PHONY: all clean
diff --git a/libuname.c b/libuname.c
index 9951b37..66ee5e1 100644
--- a/libuname.c
+++ b/libuname.c
@@ -59,7 +59,7 @@ static void *loadsym(const char *name) {
void _init(void) {
char *envstr = getenv("LIBUNAME");
char *e_uname[U_LAST] = { NULL };
-char *lasts;
+char *lasts = NULL;
register int i;
if (envstr == NULL) {
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Recording build path in .buildinfo
On 11/19/2014 07:42 AM, Jérémy Bobbio wrote: > While I still think it would be a good idea to write these patches and > push for a canonical build location, I am now thinking that there's a > way to be a bit more flexible. If we would record the build path as part > of the environment in the .buildinfo, the (yet-to-be-written) `srebuild` > script could unpack the source in that location and simply build it from > there. on the one hand, i'm a bit leery about allowing srebuild to automatically place the build tree at any arbitrary location. what if .buildinfo says "oh yeah, unpack this thing into /etc" (or /root or /usr)? otoh, the .buildinfo also says "install these versions of these packages" so maybe it's just not that big a deal. (and srebuild could also just decline to unpack things in particularly ugly locations) On the third hand, this doesn't really help the gdb use case at all :/ Ultimately, i don't think i see any real harm in including the build path in .buildinfo -- certainly once we get around to finally making things always build in the canonical build path, then great! everyone's will look the same. And if including the build path in .buildinfo unblocks the current process at all, it's probably worth doing. --dkg signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
