Bug#884095: flag to force file types
Chris Lamb: > Hi Hans, > >> It would be literally impossible to auto-detect since a Janus APK is >> both a valid DEX file (starting with the bytes "dex") and […] > > Oh dear, I got a little lost in the weeds of Janus/APK/ZIP here.. > > Could you excuse my pedanticness and ask for direct links to files, > what you are seeing and what you are expecting? That would immediately > clarify a few questions and avoid a lengthy back-and-forth :) > > > Best wishes, https://www.androidpolice.com/wp-content/uploads/janus-poc/HelloWorld-Janus.apk Or create your own: https://github.com/odensc/janus .hc ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#884095: flag to force file types
Chris Lamb: > Hi Hans! > >>> Have we really exhausted the detection route for this? :) >> >> I think the detection route has been exhausted. It seems that no one >> wants to do what it takes to reliably detect APKs. > > I'm sorry you think so and, with the greatest of respect, I'm not > sure this is entirely accurate... at least from my point of view. > > Could you perhaps attach or otherwise link to some testcases where > diffoscope gets the detection wrong? It sounds like a fun challenge, > if nothing else.. Any Janus APK, including the examples linked to in the github, etc. are test cases. It would be literally impossible to auto-detect since a Janus APK is both a valid DEX file (starting with the bytes "dex") and a functional ZIP and APK. Most ZIP readers will happily skip any bytes that don't make sense before the ZIP contents, since the file information is stored at the end of a ZIP. A Janus APK is technically not an officially valid ZIP, since it has non-ZIP bytes before the ZIP header. The most recent APK tools now reject Janus APKs as invalid, but zip tools will still happily work with them. So in my case, I'd want to compare a valid APK with a modified version of the valid APK that turns it into a Janus APK. As for increasing the reliability of the auto-detection, I think libfile could do a quick check for APK Signature v2 or v3, then reliably mark the file as an APK (vs. ZIP or JAR). APK Signature v2: https://source.android.com/security/apksigning/v2 APK Signature v3: https://android-review.googlesource.com/c/platform/tools/apksig/+/587834/ .hc ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#884095: flag to force file types
Chris Lamb: > severity 884095 wishlist > thanks > > Hi hc, > >> Something like --force=apk would solve both. > > So, I'm a little nervous about introducing such a directive. > > This is primarily in terms that diffoscope should really just Do The > Right Thing by default in all cases and not need magic flags to get a > the desired result. :) > > This is just a better user experience but also has real practical > implications; it is not tidy (or even possible) to specify such flags > in automated or hosted CI environments such as tests.reproducible-builds.org, > try.diffoscope.org. Travis CI, etc. on a per-package basis. > > Whilst we might have other flags that you could point to that would > violate this informal "rule", I would certainly cheer their removal. > > (There are also — entirely secondary — concerns around whether this > flag would change the behaviour in nested files as well, but we can > leave that for now..) > > Have we really exhausted the detection route for this? :) > > > Regards, > I think the detection route has been exhausted. It seems that no one wants to do what it takes to reliably detect APKs. I understand why libfile does not want to include more elaborate checks like: * ZIP file * with AndroidManifest.xml in it There are also often cases when working with malware samples that they are deliberately created to avoid being detected as APKs, for example the "Janus" vuln https://github.com/odensc/janus. That works by making the APK seem like a DEX file. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#890904: diffoscope does not show classes.dex diff
sorry, the server was down. Its back up so you should be able to access those links now. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#890904: diffoscope does not show classes.dex diff
Package: diffoscope Version: 90~bpo9+1 Attached are two APKs that have different classes.dex files. They are the same size, but have different contents. diffoscope does not show a diff for them. When I extract the classes.dex files from the APK, diff and vbindiff do show the differences. Here are the test files: https://verification.f-droid.org/tmp/a2dp.Vol_137.apk https://verification.f-droid.org/tmp/sigcp_a2dp.Vol_137.apk And the report: https://verification.f-droid.org/tmp/a2dp.Vol_137.apk.diffoscope.txt https://verification.f-droid.org/tmp/a2dp.Vol_137.apk.diffoscope.html ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#884095: flag to force file types
Package: diffoscope Version: 88 The Janus bug for Android works by making a valid APK file that is also a valid DEX file. https://www.guardsquare.com/en/blog/new-android-vulnerability-allows-attackers-modify-apps-without-affecting-their-signatures Diffoscope sees these files as different file types, so there is no way to imspect the malware payload. Given this and the issues in file detection in #849782, there should be a way to force which kind of comparison that diffoscope does. Something like --force=apk would solve both. There are two example files attached. HelloWorld.apk Description: application/vnd.android.package-archive HelloWorld-Janus.apk Description: application/vnd.android.package-archive signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#875451: affects 86
Just tried with diffoscope 86, same thing:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 401, in
main
sys.exit(run_diffoscope(parsed_args))
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 360, in
run_diffoscope
Config().check_constraints()
File "/usr/lib/python3/dist-packages/diffoscope/config.py", line 62,
in check_constraints
self.check_ge("max_diff_block_lines", "max_page_diff_block_lines")
File "/usr/lib/python3/dist-packages/diffoscope/config.py", line 59,
in check_ge
raise ValueError("{0} ({1}) cannot be smaller than {2}
({3})".format(a, va, b, vb))
ValueError: max_diff_block_lines (100) cannot be smaller than
max_page_diff_block_lines (128)
$ diffoscope --version
diffoscope 86
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#875451: diffoscope crashes when using --max-diff-block-lines
Package: diffoscope
Version: 85~bpo9+1
`fdroid verify` calls diffoscope like this:
diffoscope --max-report-size 12345678 \
--max-diff-block-lines 100 \
--html foo.html --text foo.txt \
foo.apk another_foo.apk
And it has recently started to crash like this:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 396, in
main
sys.exit(run_diffoscope(parsed_args))
File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 356, in
run_diffoscope
Config().check_constraints()
File "/usr/lib/python3/dist-packages/diffoscope/config.py", line 62,
in check_constraints
self.check_ge("max_diff_block_lines", "max_page_diff_block_lines")
File "/usr/lib/python3/dist-packages/diffoscope/config.py", line 59,
in check_ge
raise ValueError("{0} ({1}) cannot be smaller than {2}
({3})".format(a, va, b, vb))
ValueError: max_diff_block_lines (100) cannot be smaller than
max_page_diff_block_lines (128)
Since we're not setting max_page_diff_block_lines, this should not crash.
.hc
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#868486: diffoscope often fails to detect APKs
The APK format is a ZIP file that always includes the files AndroidManifest.xml and classes.dex. Then it also always has a JAR signature (i.e. META-INF/). It does not have the JAR magic number CAFEBABE in it. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#868486: updated URL
I had to move this APK to here: https://verification.f-droid.org/logs/Zom-15.1.0-alpha-5-zomrelease-release-unsigned.apk ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#868486: diffoscope often fails to detect APKs
Package: diffoscope Version: 83 APKs are basically a ZIP file with a JAR signature, but not necessarily the CAFEBABE byte sequence that marks a JAR. This means that comparing APKs with diffoscope often results in a straight binary diff, which is useless. Here's one example: https://verification.f-droid.org/im.zom.messenger_1510005.binary.apk.diffoscope.html im.zom.messenger_1510005.binary.apk is available here: https://verification.f-droid.org/Zom-15.1.0-alpha-5-zomrelease-release-unsigned.apk im.zom.messenger_1510005.apk is available here: https://github.com/zom/Zom-Android/releases/download/15.1.0-alpha-5/Zom-15.1.0-alpha-5-zomrelease-release.apk You can get lots and lots of APKs from here: https://f-droid.org/packages I'd like a way to force the file type in diffoscope. We are calling it from a build process, so we already know all files are going to be APKs. Also, I tried to get this added to libfile, but upstream is not willing to accept detection routines that rely on more complicated things like presence of a file in a ZIP. They just want byte patterns, which is not enough to consistently detect APKs. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#851147: --max-report-size does not apply to --text reports
Package: diffoscope Version: 67 On https://verification.fdroid.org, diffoscope is run like this: diffoscope --max-report-size 12345678 --max-diff-block-lines 100 \ --html foo.html --text bar.txt The HTML reports are being size-limited, but there are still some giant text reports, including a few that are hundreds of MB: https://verification.f-droid.org/?C=S;O=D ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#850501: rerunning using diffoscope 67-21-gfe7ae15
Thanks for your work on the APK diffing! I had to fix a typo to get it running that was introduced in diffoscope commit fe7ae15e1c177866acd478af4cc4a51bd5002017 at the bottom of it. It turned 'f_out' into a non-existent 'w'. With that change, diffoscope is now working for me again. I'm running it on the fdroid verification server, you can see the output here: https://verification.f-droid.org/ Also, about apktool.yml, I don't think we need it at all in diffoscope. It is not a file from the APK and it is not produced by Android SDK tools. As far as I know, it is only used by apktool if you want to use apktool to reconstruct a previously "decoded" APK. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#849638: related bug
FYI, I filed https://bugs.debian.org/849782 about APKs being inconsistently detected. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#849403: androguard
androguard can extract and convert the binary AndroidManifest.xml, its python2 and already in Debian. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#849638: diffoscope 66 does binary diff on APK files
Reiner Herrmann: > On Thu, Dec 29, 2016 at 12:41:16PM +0100, Hans-Christoph Steiner wrote: >> When running diffoscope on two APKs using version 66, it now just does a >> straight binary comparison of the direct file itself. Running >> diffoscope 64 generated a nice output of the individual files in the ZIP >> (an APK is a signed JAR with some other special features). Attached is >> the output from v66. You can find v64 output here: >> >> http://37.218.242.117/ >> >> You can download these two APKs to test with from here: >> >> http://37.218.242.117/aarddict.android_26.apk >> https://f-droid.org/repo/aarddict.android_26.apk > > Just had a short look at this. The reason is that file/libmagic detects > the files differently: > > $ file 1/aarddict.android_26.apk 2/aarddict.android_26.apk > 1/aarddict.android_26.apk: Java archive data (JAR) > 2/aarddict.android_26.apk: Zip archive data, at least v2.0 to extract > > So the APK comparator should probably also work on files detected as zip. > Yeah, that makes sense as long as libmagic cannot reliably detect APKs vs JAR vs ZIP. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#849638: downgrading back to 64 gives me the same
So it seems that the issue is not in diffoscope per se, since now downgrading back to 64 from snapshot.debian.org generates the same output. I'm guessing then this is related to interactions with the dependencies, since I also did an `apt upgrade` at the same time. This is on a machine running stretch. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Bug#849638: diffoscope 66 does binary diff on APK files
Package: diffoscope Version: 66 Severity: important When running diffoscope on two APKs using version 66, it now just does a straight binary comparison of the direct file itself. Running diffoscope 64 generated a nice output of the individual files in the ZIP (an APK is a signed JAR with some other special features). Attached is the output from v66. You can find v64 output here: http://37.218.242.117/ You can download these two APKs to test with from here: http://37.218.242.117/aarddict.android_26.apk https://f-droid.org/repo/aarddict.android_26.apk aarddict.android_26.apk.html.bz2 Description: application/bzip ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
Daniel Kahn Gillmor: > On Mon 2015-02-16 04:39:32 -0500, Hans-Christoph Steiner wrote: >> I think this topic is far too vast with far too many dependencies to really >> have a useful discussion on without a full time, dedicated team. Since that >> seems highly unlikely in the near future, we need to break it down into >> chunks >> of work that we can achieve with the time and resources we actually have. >> >> So we need to focus on drilling down to what is the simplest useful form of >> package signing that will cause the least amount of problems when we decide >> to >> change how package signing works. This means we get a prototype out as soon >> as possible, and we can learn a lot from that. I think that's pretty easy to >> do, something like this: >> >> * make dpkg optionally check package sigs, and refuse to install on bad sig >> * use apt signing model: signatures verified from the apt key ring >> * signing can start happening in the build tools, by the uploader >> * start work towards getting the Debian built/apt infrastructure signing > > The above bullet points don't seem mutually compatible. if the uploader > is doing the per-package signing, then their key won't be in the apt > keyring; this would make each package have a bad sig; and would mean > that dpkg in signature-checking mode would reject the packages. > > Choosing the apt keyring as the signing model appears to punt entirely > on questions of corroborative authority and how that plays out in a > multi-distro ecosystem. > > But overall, I think this discussion of embedded per-package signatures > misses the point: the only reason we're talking about possible future > per-package signatures is because we want to avoid breaking them as we > add mechanisms to support reproducible builds. > > We *have* a team actively working on reproducible builds (this mailing > list), who have done a lot of work already and have thought about the > needs for that project. > > If we don't have a use case or a plan or a team or a goal or known > constraints for doing embedded per-package signing, then delaying the > archive and system changes needed by reproducible builds on behalf of > some non-concrete desire for per-package signatures is a mistake. We > shouldn't be blocking progress on an active and effective project for > one that isn't underway at all. > > And remember that we already have a very clear understanding of how to > do *non-embedded* per-package signatures, should anyone want to do them. > > My pkg-fingerprint proposal was trying to lay out compromise that would > leave per-package embedded signatures possible in the future without > having to specify them in detail today. I don't *want* to specify them > in detail today. I want to be able to move on with making the archive > reproducible :) > > --dkg I cannot imagine an development effort in the reasonable future that would actually achieve some kind of multiple signature tricks for embedded sigs. I think the only reasonable thing to consider is a single canonical embedded signature, because that is something that can actually be implemented, technically, time-wise, and politics-wise. It is also something that is widely implemented in other systems, like Android. Its a tried-n-true idea. And a single-canonical embedded signature is actually not hard when it comes to building reproducibly. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 https://pgp.mit.edu/pks/lookup?op=vindex&search=0x9F0FE587374BBE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
Daniel Kahn Gillmor: > On Fri 2015-02-13 03:36:20 -0500, Hans-Christoph Steiner wrote: >> I think it would be much simpler to just have the single package signature >> that is embedded in the package file itself, like Android APKs and Java JARs. >> Since the package is built reproducibly, anyone who builds it can just copy >> the canonical signature into their copy of the package they just built, and >> it'll match the sha512sum of the signed apt metadata. > > It seems like you're saying everyone will be able to agree on which > signing authority is "canonical" for any given package. I'm not > convinced that's the case. > >> The big question there is determining the where the canonical >> signature happens. It seems like it should be the official Debian >> build process, since it is the only process guaranteed to be the same > > Even though i'm personally likely to treat Debian as "the canonical > source" i care about, i don't want it to be that way. I would like > Debian to be able to be a downstream as well as an upstream (see the > work feeding back into debian from ubuntu, for example); if a .deb > package can contain an internal signature, and i'm looking at a given > .deb in isolation on my debian system, i want to see it signed *by > debian*, not by whoever happened to produce it first. Otherwise, it's > not clear to me that the embedded signature is useful to me as an end > user at all. > >> Another question is whether dpkg checks whether the signers match when >> upgrading, like the Android model (a package can only be upgraded by >> another package signed by the same key). This would be nice, but >> seems optional and hard to do in Debian. > > Maybe this is the question we need to answer to move the discussion > forward to make sure we're taking the desire for embedded signatures > into account when thinking about reproducible .debs: how exactly do we > expect an embedded signature to be used/evaluated? by who, and in what > context? > >> I think the .buildinfo file is useful, but for a separate process. It should >> be the canonical file for running a reproducible build. > > I'm not sure what this means. I'd be very happy if *all* of my debian > packages were reproducible builds, and i could have a way of verifying > it. I'd consider that more valuable than knowing that all my .debs were > signed by any individual authority. So if we're really talking about a > tradeoff between signed buildinfo files and signed packages, i'd > certainly prefer signed buildinfo files. > > But my proposal was an attempt to let people have both, without forcing > the entire ecosystem to agree on "who is a canonical authority for > package X, without whom a reproducible package is impossible" > > --dkg I think this topic is far too vast with far too many dependencies to really have a useful discussion on without a full time, dedicated team. Since that seems highly unlikely in the near future, we need to break it down into chunks of work that we can achieve with the time and resources we actually have. So we need to focus on drilling down to what is the simplest useful form of package signing that will cause the least amount of problems when we decide to change how package signing works. This means we get a prototype out as soon as possible, and we can learn a lot from that. I think that's pretty easy to do, something like this: * make dpkg optionally check package sigs, and refuse to install on bad sig * use apt signing model: signatures verified from the apt key ring * signing can start happening in the build tools, by the uploader * start work towards getting the Debian built/apt infrastructure signing .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 https://pgp.mit.edu/pks/lookup?op=vindex&search=0x9F0FE587374BBE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Preliminary review of dpkg-genbuildinfo
Daniel Kahn Gillmor:
> On Fri 2015-02-06 01:13:18 -0500, Guillem Jover wrote:
>> Take the example I gave previously of a binary package detached from
>> an archive, just a .deb package laying around, either from an old
>> download or passed to you by someone. You have to *know* the origin of
>> the binary, otherwise you need to first start hunting down where this
>> binary was built, say Debian, one of its derivatives or even somewhere
>> else. And sure, once that's known, the user *might* possibly be able to
>> reproduce the build, but I don't see many (if not most) users being able
>> or willing to set up a reproducible build environment just to verify
>> where a binary was coming from (say my relatives). If you cannot or
>> wont do that, you need to trust the distribution, the remote server,
>> the network, the remote binary including any possible reproducible
>> information being correct. At that point you or a program might as well
>> have just verified an embedded signature.
>
> I see per-package signatures and reproducible builds as complementary to
> one another. I'd be happy to brainstorm ways that we can make sure we
> can get the advantages of both.
>
> I understand the goals of per-package signatures, and that it's more
> convenient for most folks to have a single file than to have to schlep
> around multiple files to be able to verify the package.
>
> I also understand the ecological benefits of reproducibility, especially
> for a project like debian that has so many heterogenous downstreams that
> pull (directly or indirectly) from it.
>
> For one of the big reproducibility benefits -- detecting compromised
> build environments -- to accrue to everyone, we only need a handful of
> independent reproducers who can't all be compromised together, and a
> straightforward way that people can compare the results and raise an
> alarm if something looks fishy.
>
> The benefits of reproducibility come from *corroboration*. otoh, the
> usual view of embedded package signatures come from *single authority*
> ("the origin of the binary"). For the broader ecosystem of debian + its
> derivatives, where some binary packages might be shipped by multiple
> vendors, it is actually useful to users of a derivative that doesn't
> rebuild everything to be able to say with confidence "this package is
> exactly the same as the packages debian ships". At the same time, as a
> user of the derivative, you'd also like to be able to know that any
> given package is "endorsed" or included by your distro, so that
> arbitrary packages from debian proper can't just be slipped by your
> package manager without knowing about it.
>
> So there's this interplay (tension?) between authority of origin and
> corroboration that i suspect we'd all like to support.
>
> I've been partial to the separate .buildinfo file specifically because i
> think i can see the path where we can get both corroboration and
> specific authority from it (a single .buildinfo file can be signed by
> multiple authorities), while the identical .deb in two distros can be
> bitwise-compared using simple tools.
>
> That said, this clearly doesn't address the convenience/portability
> win of embedded package signatures raised by Guillem.
>
> As a step toward brainstorming, here's a way that we could (maybe?) get
> both:
>
> * we could make a pkg-fingerprint tool that takes a package and
>produces a cryptographically-strong fingerprint of the contents of
>the package *minus* the signature elements.
>
>As long as the package format has no internal signature, this would
>just be something like sha512sum over the entire package. When we
>have a package with an embedded-signature format, we'd need to define
>a way to strip any/all signatures from the package in a reproducible
>way that does not touch the rest of the contents of the package.
>then pkg-fingerprint becomes something like:
>
> pkg-strip-sigs < foo.deb | sha512sum
>
>and the comparison between packages moves from /usr/bin/cmp to
>comparing the outputs of pkg-fingerprint (or we could make a pkg-cmp
>tool).
>
>the buildinfo files would then store the pkg-fingerprint output over
>the .deb or .rpm binaries that they produced.
>
> This is a little bit inelegant, but maybe it's a way that we can have
> our cake and eat it too? I'd love to hear other suggestions.
>
> --dkg
I think it would be much simpler to just have the single package signature
that is embedded in the package file itself, like Android APKs and Java JARs.
Since the package is built reproducibly, anyone who builds it can just copy
the canonical signature into their copy of the package they just built, and
it'll match the sha512sum of the signed apt metadata. The big question there
is determining the where the canonical signature happens. It seems like it
should be the official Debian build process, since it is the only process
guaranteed to be the same
Anoth
Re: [Reproducible-builds] Wiki reorganization
Looks drastically better! I think this wiki is really the central resource for anyone interested in making reproducible builds, Debian or not. So I'm glad to see it reorganized to look like a community resource rather than the giant notepad it was before. .hc Jérémy Bobbio: > Hi! > > While waiting on builds and rebuilds of linux, I started to reorganize, > refresh and improve the documentation on the wiki. > > .O. > === HEADS-UP! ==o===o= HEADS-UP! === > > If you were previously subscribed to the ReproducibleBuilds wiki > page, you should edit your notifications: > > https://wiki.debian.org/?action=userprefs&sub=notification > > And add to the list of pages the following regex: > > ReproducibleBuilds/.* > > Otherwise, you are likely to miss some of the fun! > > > > The main page is now a landing page with links based on potential > reader's interests. Have a look: > https://wiki.debian.org/ReproducibleBuilds > > I'm happy with the status of the About, Contribute, and > ExperimentalToolchain sub-pages. I should be able to work on Howto > and History tomorrow. Don't wait for me, though. :) > > > > ___ > Reproducible-builds mailing list > Reproducible-builds@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds > -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 https://pgp.mit.edu/pks/lookup?op=vindex&search=0x9F0FE587374BBE81 signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [RFC] debbindiff
Comparing jars gives a stacktrace, looks like a missing import.
$ ./debbindiff.py
~/code/guardianproject/cacheword/cachewordlib/cachewordlib-v0.1-1-g04cb18e.jar
/tmp/cachewordlib-v0.1-1-g04cb18e.jar
Traceback (most recent call last):
File "./debbindiff.py", line 53, in
main()
File "./debbindiff.py", line 43, in main
differences = debbindiff.comparators.compare_files(parsed_args.file1,
parsed_args.file2)
File
"/media/share/code/reproducible/debbindiff/debbindiff/comparators/__init__.py",
line
85, in compare_files
return comparator(path1, path2, source)
File
"/media/share/code/reproducible/debbindiff/debbindiff/comparators/utils.py",
line 51, in with_fallback
inside_differences = original_function(path1, path2, source)
File
"/media/share/code/reproducible/debbindiff/debbindiff/comparators/zip.py",
line 57, in compare_zip_files
zipinfo1 = get_zipinfo(path1)
File
"/media/share/code/reproducible/debbindiff/debbindiff/comparators/zip.py",
line 31, in get_zipinfo
return re.sub(re.escape(path), os.path.basename(path), output)
NameError: global name 're' is not defined
Also, I updated the setup.py for two small things. I recommend using code
checkers like pyflakes and pylint:
$ pyflakes *.py debbindiff/*.py
debbindiff/difference.py:20: 'difflib' imported but unused
debbindiff/difference.py:41: redefinition of function 'comment' from line 37
hans@palatschinken debbindiff $ pylint *.py debbindiff/*.py
...
.hc
Hans-Christoph Steiner wrote:
>
> Definitely use setup.py. It makes the packaging easy and standardized, and it
> is the standard way to build python. It also makes it easy to publish
> releases to pypi, the central package repository for python. I attached a
> quick untested one for you.
>
> .hc
>
> Jérémy Bobbio wrote:
>> Hi!
>>
>> I've been working at high pace since Sunday on a replacement for the
>> diffp script [1]. These GPLv3 lines of Python are called debbindiff.
>>
>> Get it from Git:
>>
>> https://anonscm.debian.org/cgit/reproducible/debbindiff.git/
>>
>> Attached is an output produced for the attr package. The new tool is at
>> least as capable as diffp, is way more extensible, and the result is
>> more readable.
>>
>> Example usage:
>>
>> $ ./debbindiff.py --html /tmp/debbindiff.html b1/*.changes b2/*.changes
>>
>> There's no requirements for actually comparing .changes. You can use it
>> to compare jar files directly if that's your kick.
>>
>> I'd love to see reviews of the code. It's scarce on comments but names
>> should be explicit enough, or so I hope.
>>
>> It's missing Debian packaging. I guess I should learn how to write a
>> setup.cfg or similar. Pointers or patches welcome.
>>
>> One thing this codebase should enable is writing “hints”. Once the tree
>> of differences is generated, it should be doable to run through it to
>> generate statements like: “Many files in data.tar have different
>> timestamps, dh_fixmtimes has probably not been called. Are you
>> using dh?” This still needs to be done though.
>>
>> Last note: I've been pushing everything else aside while I had the
>> thrills to work on this. It's unclear when will be the next time, so
>> patches are preferred rather than suggestion.
>>
>> [1]: https://anonscm.debian.org/cgit/reproducible/misc.git/tree/diffp
>>
>>
>>
>> ___
>> Reproducible-builds mailing list
>> Reproducible-builds@lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
>>
>
--
PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81
#!/usr/bin/env python2
from setuptools import setup
setup(name='debbindiff',
version='0.1',
description='display differences between files',
long_description=open('README').read(),
author='Lunar',
author_email='lu...@debian.org',
url='https://wiki.debian.org/ReproducibleBuilds',
packages=['debbindiff'],
scripts=['debbindiff.py'],
install_requires=[
'python-debian',
],
classifiers=[
'Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
'Operating System :: POSIX',
'Topic :: Utilities',
],
)
signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [RFC] debbindiff
Definitely use setup.py. It makes the packaging easy and standardized, and it
is the standard way to build python. It also makes it easy to publish
releases to pypi, the central package repository for python. I attached a
quick untested one for you.
.hc
Jérémy Bobbio wrote:
> Hi!
>
> I've been working at high pace since Sunday on a replacement for the
> diffp script [1]. These GPLv3 lines of Python are called debbindiff.
>
> Get it from Git:
>
> https://anonscm.debian.org/cgit/reproducible/debbindiff.git/
>
> Attached is an output produced for the attr package. The new tool is at
> least as capable as diffp, is way more extensible, and the result is
> more readable.
>
> Example usage:
>
> $ ./debbindiff.py --html /tmp/debbindiff.html b1/*.changes b2/*.changes
>
> There's no requirements for actually comparing .changes. You can use it
> to compare jar files directly if that's your kick.
>
> I'd love to see reviews of the code. It's scarce on comments but names
> should be explicit enough, or so I hope.
>
> It's missing Debian packaging. I guess I should learn how to write a
> setup.cfg or similar. Pointers or patches welcome.
>
> One thing this codebase should enable is writing “hints”. Once the tree
> of differences is generated, it should be doable to run through it to
> generate statements like: “Many files in data.tar have different
> timestamps, dh_fixmtimes has probably not been called. Are you
> using dh?” This still needs to be done though.
>
> Last note: I've been pushing everything else aside while I had the
> thrills to work on this. It's unclear when will be the next time, so
> patches are preferred rather than suggestion.
>
> [1]: https://anonscm.debian.org/cgit/reproducible/misc.git/tree/diffp
>
>
>
> ___
> Reproducible-builds mailing list
> Reproducible-builds@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
>
--
PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81
#!/usr/bin/env python2
from setuptools import setup
import sys
setup(name='debbindiff',
version='0.1',
description='display differences between files',
long_description=open('README').read(),
author='Lunar',
author_email='lu...@debian.org',
url='https://wiki.debian.org/ReproducibleBuilds',
packages=['debbindiff'],
scripts=['debbindiff'],
install_requires=[
'python-debian',
],
classifiers=[
'Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
'Operating System :: POSIX',
'Topic :: Utilities',
],
)
signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Stefan Fritsch wrote: > On Sunday 21 September 2014 21:13:50, Richard van den Berg wrote: >> Package formats like apk and jar avoid this chicken and egg problem >> by hashing the files inside a package, and storing those hashes in >> a manifest file. Signatures only sign the manifest file. The >> manifest itself and the signature files are not part of the >> manifest, but are part of the package. So a package including it's >> signature(s) is still a single file. > > This is bad design and will inevitably lead to security issues (as has > been demonstrated by Android and apk). One must check the signature > first, and only if the signature matches, start parsing complex file > formats. And yes, zip is complex enough to be a problem. It is true that an embedded signature requires more complicated code, but it also simplifies the parts that the user has to understand. Perfect code with a bad user experience will also inevitably lead to security issues. I'm guessing that ar format is simpler than zip, so that'd be helpful. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > On 09/22/2014 04:06 PM, Hans-Christoph Steiner wrote: >> I think we're starting to nail down the moving parts here, so I want to >> outline that so we can find out the parts where we agree and where we >> disagree. >> >> * I hope we can all agree that the package itself should not change once it >> has hit the official repos. >> >> * I believe we can achieve what we want without taking a shortcut and >> introducing a new core package type (.sdeb .debs or whatever). We can figure >> out how to do this with the .deb file. Personally, I would accept a new >> package type after a thorough exploration of keeping .deb fails to deliver, >> but not before. >> >> * There should be at least one verification build before a package becomes >> official. >> >> * Then there needs to be a channel for people to submit the results of their >> own builds. That could be only positive results or only negative results, or >> both. >> >> * the .buildinfo file should contain all info needed to reproduce the build, >> given a standard Debian build environment > > Thanks, the above is a very useful summary. > >> Anything I left out? > > I think the summary above hints at but doesn't answer the question of > what an "official" package means, and the fact that there may be > multiple repositories (possibly operated by different organizations) > with different rules about what should make a package "official". > > I think we need to ask whether we care about byte-for-byte identical > .deb files *across* different repositories or not. > > If we don't care about cross-repo (or cross-organization) byte-for-byte > reproducibility, then an embedded signature in the .deb might be > acceptable (though the data it contains would be redundant to signatures > over the buildinfo files, which would eventually be necessary for > external policies or corroboration anyway). > > If we *do* decide that we care about cross-repo byte-for-byte > compatibility, then embedding a signature in the .deb suggests that one > repo can act as the gating factor for another, because repos > collaborating in this reproducibility push cannot both hold the key that > makes a .deb "official". > > I don't think that's a good tradeoff. As tempting as it might be to try > to cement debian's "authoritative" role via such a lock-in, i'd much > rather than debian derivatives, blends, side projects, etc, can all take > initiative that can then be absorbed back into debian cleanly and > reproducibly. > > i also suspect that the redundancy between internal signatures and > signed .buildinfo records is likely to cause some increase in confusion, > but i don't think that's as serious of a problem as the question of > which signing keys get to be "authoritative". > > --dkg Cross-repo byte-for-byte compatibility is a nice thing to strive for, but it sounds quite difficult to achieve and will require lots of social coordination as well as technical work. In terms of builds of a particular .deb by multiple distros, each distro will have to use the exact same toolchain to build the .deb for most packages. Different versions of gcc, javac, etc. will produce different binaries. You'll have the same problems as the canonical signature, like if two distros make a new package at the same time, but with different standards (gcc version, signer, etc). Ubuntu's gcc version will create a .deb with one hash, and Debian's gcc will create a .deb with a different hash, and each distro will mark theirs as canonical. That seems to be a much harder thing to manage across the distros. So if another distro or repo is going to buy into Debian's reproducible system, adding a bit about canonical signatures seems totally feasible to manage. The canonical signature would just need to be done by a key in the debian-keyring for Debian, ubuntu-keyring for Ubuntu, etc. While a static canonical signer is desirable, I don't think it can be supported without adding some restrictions (I think it would be worth it, for the record). Whoever is the first to publish a given package version claims the canonical role for both build setup and signature. To prevent accidental collisions, dput could check the various NEW queues, the various package repos, etc. to look for an existing canonical package. Then the first distro to publish is canonical. Shall we have a real time discussion on this topic? voice, video, or in person all work for me. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > On 09/22/2014 04:06 PM, Hans-Christoph Steiner wrote: >> I think we're starting to nail down the moving parts here, so I want to >> outline that so we can find out the parts where we agree and where we >> disagree. >> >> * I hope we can all agree that the package itself should not change once it >> has hit the official repos. >> >> * I believe we can achieve what we want without taking a shortcut and >> introducing a new core package type (.sdeb .debs or whatever). We can figure >> out how to do this with the .deb file. Personally, I would accept a new >> package type after a thorough exploration of keeping .deb fails to deliver, >> but not before. >> >> * There should be at least one verification build before a package becomes >> official. >> >> * Then there needs to be a channel for people to submit the results of their >> own builds. That could be only positive results or only negative results, or >> both. >> >> * the .buildinfo file should contain all info needed to reproduce the build, >> given a standard Debian build environment > > Thanks, the above is a very useful summary. > >> Anything I left out? > > I think the summary above hints at but doesn't answer the question of > what an "official" package means, and the fact that there may be > multiple repositories (possibly operated by different organizations) > with different rules about what should make a package "official". > > I think we need to ask whether we care about byte-for-byte identical > .deb files *across* different repositories or not. > > If we don't care about cross-repo (or cross-organization) byte-for-byte > reproducibility, then an embedded signature in the .deb might be > acceptable (though the data it contains would be redundant to signatures > over the buildinfo files, which would eventually be necessary for > external policies or corroboration anyway). > > If we *do* decide that we care about cross-repo byte-for-byte > compatibility, then embedding a signature in the .deb suggests that one > repo can act as the gating factor for another, because repos > collaborating in this reproducibility push cannot both hold the key that > makes a .deb "official". > > I don't think that's a good tradeoff. As tempting as it might be to try > to cement debian's "authoritative" role via such a lock-in, i'd much > rather than debian derivatives, blends, side projects, etc, can all take > initiative that can then be absorbed back into debian cleanly and > reproducibly. > > i also suspect that the redundancy between internal signatures and > signed .buildinfo records is likely to cause some increase in confusion, > but i don't think that's as serious of a problem as the question of > which signing keys get to be "authoritative". > > --dkg Cross-repo byte-for-byte compatibility is a nice thing to strive for, but it sounds quite difficult to achieve and will require lots of social coordination as well as technical work. In terms of builds of a particular .deb by multiple distros, each distro will have to use the exact same toolchain to build the .deb for most packages. Different versions of gcc, javac, etc. will produce different binaries. You'll have the same problems as the canonical signature, like if two distros make a new package at the same time, but with different standards (gcc version, signer, etc). Ubuntu's gcc version will create a .deb with one hash, and Debian's gcc will create a .deb with a different hash, and each distro will mark theirs as canonical. That seems to be a much harder thing to manage across the distros. So if another distro or repo is going to buy into Debian's reproducible system, adding a bit about canonical signatures seems totally feasible to manage. The canonical signature would just need to be done by a key in the debian-keyring for Debian, ubuntu-keyring for Ubuntu, etc. While a static canonical signer is desirable, I don't think it can be supported without adding some restrictions (I think it would be worth it, for the record). Whoever is the first to publish a given package version claims the canonical role for both build setup and signature. To prevent accidental collisions, dput could check the various NEW queues, the various package repos, etc. to look for an existing canonical package. Then the first distro to publish is canonical. Shall we have a real time discussion on this topic? voice, video, or in person all work for me. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] faketime by default?
Jérémy Bobbio wrote: > Hans-Christoph Steiner: >> I've been using faketime in my work on reproducible builds on Android (both >> "native" C code and Java), and it has been working well. It seems to me that >> the current approach in Debian does not use faketime. >> >> Since so much of the little issues are due to timestamps, it could make a lot >> of sense if faketime was a default part of the process. For the make+C >> builds >> I've done, faketime just froze the clock entirely and that worked. For >> ant+java builds, it would hang forever with a frozen clock. So I set >> faketime >> to run the clock at 5% speed, and that seems to work well. > > Before getting into any technical details, let's start by the social > problem. We are talking about Debian (and derivatives!). There is no > canonical way a Debian package is built. Most developers will, for > example, use `fakeroot`, but it's optional and `sudo` is also supported. > To reproduce builds using libfaketime, you need both the original build > and the later builds to use libfaketime. Requiring that every Debian > developers uses libfaketime when building a package is a change that > would be very hard, if not impossible, to achieve. > > For the most important technical problem: libfaketime cannot make the > build process entirely deterministic. According to my tests, parallel > execution, variations in file order, and randomness can all participate > in making the faked times different from one build to the next. > > While experimenting in my little corner, I also saw several build > systems just fail, or fail to be reproducible with libfaketime. That's > what actually led me to remove the time fixing feature of the > `test-reproducibility` script entirely: > https://anonscm.debian.org/cgit/reproducible/misc.git/commit/?id=f4146ee73d > I really would love to have it back, but it simply could not be made to > work reliably under my fingers. > > There's a consensus that libfaketime is not the good approach for > reproducible builds since the initial BoF during DebConf13. Patching > timestamps is a harder path, but we know there's not going to be a > hidden cliff in the middle of the journey. Thanks for the explanation, that's a bummer that it was so problematic, but I suppose not surprising. Would it make sense to have something like a "dh --use-faketime" flag so that maintainers could opt-in to use it where it might be helpful like when upstream refuses to remove build stamps, etc? The idea there would be that dh would handle setting up faketime before the upstream build system was called. In that case, a flag to dh would be a lot less work to maintain then a set of patches. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Elmar Stellnberger wrote: > Am 22.09.14 um 01:52 schrieb Paul Wise: >> On Mon, Sep 22, 2014 at 2:04 AM, Elmar Stellnberger wrote: >> >>> A package with some new signatures added is no more the old package. >> That is exactly what we do *not* want for reproducible builds. >> >>> It should have a different checksum and be made available again for update. >> The Debian archive does not allow files to change their checksum, so >> every signature addition requires a new version number. That sounds >> like a bad idea to me. > Yes, that is something we definitely do not want. > Nonetheless it would still be an issue to have the package and the signatures > in one file because we usually need them together. My only idea to realize > this > in spite of the said objection would be another proposal: > Put the .deb and the signatures into one .ar called .sdeb and make tools like > dpkg work on .sdebs or on .deb + signatures respecively. Whenever someone > offers some packages for download that will be in the form of .sdebs while > official debian repositories may separate both kinds of files. User interfaces > like http://debtags.debian.net/search/ could then generate .sdebs on the fly > to satisfy petted users. I think we're starting to nail down the moving parts here, so I want to outline that so we can find out the parts where we agree and where we disagree. * I hope we can all agree that the package itself should not change once it has hit the official repos. * I believe we can achieve what we want without taking a shortcut and introducing a new core package type (.sdeb .debs or whatever). We can figure out how to do this with the .deb file. Personally, I would accept a new package type after a thorough exploration of keeping .deb fails to deliver, but not before. * There should be at least one verification build before a package becomes official. * Then there needs to be a channel for people to submit the results of their own builds. That could be only positive results or only negative results, or both. * the .buildinfo file should contain all info needed to reproduce the build, given a standard Debian build environment Anything I left out? .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] faketime by default?
I've been using faketime in my work on reproducible builds on Android (both "native" C code and Java), and it has been working well. It seems to me that the current approach in Debian does not use faketime. Since so much of the little issues are due to timestamps, it could make a lot of sense if faketime was a default part of the process. For the make+C builds I've done, faketime just froze the clock entirely and that worked. For ant+java builds, it would hang forever with a frozen clock. So I set faketime to run the clock at 5% speed, and that seems to work well. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#762397: libgpg-error: please do not capture the current time during the build process
Another option is building with faketime. I've had good luck with a couple of C builds using faketime to freeze time entirely during the whole make process. But I think the reproducible work in Debian so far has avoided using faketime. signature.asc Description: OpenPGP digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Jérémy Bobbio wrote: > Hans-Christoph Steiner: >>> This makes `.deb` hard to use without a repository for anything >>> substantial. I would assume that's why Ubuntu developed the Click >>> package format. >> >> Check out apt-offline, it makes this process easy. > > I know about apt-offline. But that was basically my point: you don't > manipulate `.deb` by themselves easily as I've seen people do with APKs. > It's external tools which make it easy. > > You were saying that we needed verifications to be as transparent and > automatic as possible. I agree. We have tools which make it happen > instead of asking for raw low-level interfaces. But the apt signatures expire after two weeks. So that does not work for an offline system. >> But .buildinfo is not a replacement for the embedded signature with an >> immutable signature. They solve different problems. This embedded signature >> idea is not really directly related to reproducible builds, but dkg started >> this thread here so I responded. > > Except that embedded signatures break the idea of independently > reproducible builds. It means that on top of a description of the build > environment and the source code, I now need to retrieve a digital > signature from the original build if I want it to match. It really does not break reproducible builds, if implemented properly. Just include the complete canonical signature in .buildinfo in base64 format. I don't see the problem there. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Jérémy Bobbio wrote: > Hans-Christoph Steiner: >> I still strongly disagree. Very very few people care enough to learn a >> separate process. For security to be usable, it needs to be as transparent >> and automatic as possible. APKs and Android have demonstrated that you can >> have this kind of system working well. > > Comparing .deb and APKs is misleading when talking of tools. A given > .deb will have dependencies. APKs are self-contained. That is only true to a small degree. Many APKs depend on other APKs to provide "Activity" or "Service" instances that respond to "Intent" messages or are accessible via "AIDL" interfaces. For example, if your app wants to interact with OpenKeychain as a OpenPGP provider, then OpenKeychain must be installed. Android is fundamentally built around the idea that apps rely on each other provide chunks of functionality. It is true that APKs basically never use shared libraries from other APKs, but that is largely because Android is a very different OS than UNIX, Windows, etc. If you are thinking of Android as some kind of Linux distro, you will be quite misguided in understanding how it actually works. > This makes `.deb` hard to use without a repository for anything > substantial. I would assume that's why Ubuntu developed the Click > package format. Check out apt-offline, it makes this process easy. That's just one example. The problem is that if the apt cache is more than two weeks old, you can no longer even install .deb files that were previously verified and stored in the apt cache. This is why I'm interested in signatures embedded in the .deb as something separate from the apt signatures. I also regularly download individual .deb packages when I need a to use a newer one in my old stable setup. That is vastly easier to use than apt preferences, even when considering that I have to manually check the checksum. >> They've made the whole process easier by requiring the upstream >> developer be the manager of the signing. I think setting up a similar >> role in Debian will be quite beneficial, and dak and the package >> maintainer are natural roles to be the signer. > > With the current .buildinfo signing scheme, we require the Debian > maintainer to provide a package that can be built reproducibly. Then we > can require a proof of that reproducibility from the maintainer, any > other maintainers, and any number of buildds. These assessments that a > build can be properly reproduced can come after the initial upload. We > can only do that if the .deb files do not change after they hits the > archive. .buildinfo sounds great for managing the reproducible build process and an unlimited number of build verification signatures, that sounds fine by me. Then anyone who wants to verify the build themselves can use the tools for that process, and those tools can handle the .buildinfo file without the user having to know about them. But .buildinfo is not a replacement for the embedded signature with an immutable signature. They solve different problems. This embedded signature idea is not really directly related to reproducible builds, but dkg started this thread here so I responded. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > Thanks for the discussion, Hans. > > On 09/19/2014 02:47 PM, Hans-Christoph Steiner wrote: >> Packages should not be accepted into any official repo, sid included, without >> some verification builds. A .deb should remain unchanged once it is accepted >> into any official repo (maybe experimental could be an exception, but not >> sid). I think that is essential. > > But some repositories could have different rules for package inclusion > than others, right? for example, say debian wanted to offer an > unstable-reproducible suite, which only permitted packages that had been > independently rebuilt reproducibly by multiple DDs and at least two > different buildds. Ideally, the packages that are shared between this > repository and other repositories would be identical. > > Note that if .deb files are internally signed, two developers *cannot* > create the same exact .deb if they do not share their secret keys. You're missing one key detail here, let's see if I can suss it out: * the builds are _exactly_ the same, except the signatures * the embedded signature does not sign the signature files (see jar and apk formats, which are almost the same, for examples) * anyone can just copy other dev's signature into the package and it will validate because the package contents are exactly the same * the signature files sign the package contents, not the hash of whole .deb file (i.e. control.tar.gz and data.tar.gz). Therefore two developers can easily create the same .deb if that have access to the signature file since they can just copy it. No need to run the signing process again. If people create their own .deb files in a reproducible process, then copy in the same signature files, then the hash of the .deb will be the same. >> I see no reason for changing the .deb between sid and testing, except for >> perhaps how existing implementations are doing it. It is usually worth the >> work to do things right way, rather than the easy way. > > I agree with this sentiment, i think we're trying to sort out what is > the "right way". > >> The build verification process needs to happen between the package upload and >> publishing to sid or security updates. Two builds is easy: the .deb that the >> uploader generates and the one the Debian process makes. That is probably >> enough. >> >> In Debian's case, it probably is too complicated to include multiple >> signatures. In that case, there should be only one canonical signature by >> dak >> once the build verification signature threshold has been passed. Then all of >> the other signatures could be added to .buildinfo or .changes or whatever >> other file. > > but the .buildinfo file is designed to say "i generated the .deb that > matches this digest exactly", which the corroborating builder cannot do, > because they cannot produce the internal signature. No need to produce the signature, just copy it! > Plus, we now have two different places to look for signatures. one > "canonical" one and then some external ones, and the signatures > themselves have different properties (one signs parts of the deb, the > other signs the whole .deb; one signs the build environment, the other > does not, etc) Definitely look at jar signing, it handles multiple signatures fine. I see no reason why you can't include an unlimited number of signatures in a .deb. Changing the number of signatures will change the hash of the .deb, that is why there needs to be a canonical set of signatures for each .deb. As for signing the hash of the entire .deb, that is what apt already gives us, that does not need to be reproduced in the dpkg-sig embedded signature. For people who want to verify the contents of a .deb with any kind of signature, then a tool will have to compare the hashes of control.tar.gz and data.tar.gz. >> Another option is to do it like f-droid.org does. F-droid.org generates a >> APK >> signing key for each app, then manages the signing on a specialized signing >> server. Or another option is just requiring all the signers to be from the >> debian-keyring, rather than an exact match for previous signers. > > Again, i think this is getting ahead of the discussion. i'm not > proposing that we try to set debian (or other derived distro) archive > policy here, i just think we want to think > >> In any case, the .deb needs to remain unchanged. > > right. but it can't be unchanged if the archive distributor decides > that a different signer is the "canonical" signer. So you're making the > contents of the .deb dependent on archive policy, rather than the other > way around. > > I
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > Hi Hans-- > > I think we're in agreement here about most things actually, despite our > back-and-forth. hopefully this is a clarifying response: > >> Daniel Kahn Gillmor wrote: > In that case, the .deb that was installed on a sid system *is not* the > .deb that is installed on a testing system. > > If i run a mixed unstable/testing system (i do, actually, this is not > hypothetical) should i need to re-install foo_1.2-3_mipsel.deb when that > package transitions from unstable to testing (without any changes made > to it other than new signatures)? That seems odd, but the .debs are now > no longer bytewise identical. should archive operators who are doing > rsync mirroring of a number of pools update their .debs as new > signatures are added to them? can they still use rsync for this cleanly > without a massive increase in bandwidth between mirrors or do we need to > define a new synchronization mechanism? > > The question of what is a "canonical, immutable signature" for any given > distribution is also problematic, because it ties the policy of the > distribution (already defined by what that apt repository includes and > references in Release.gpg) to a set of individual package signatures. > But this is exactly the point where we'd like more flexibility. People > who care about apt repo X and use it online can use Release.gpg, while > people who are *not* using the apt repo might have a different set of > policies. And some repos might want to share specific packages with > each other -- what if their signing policies about the "canonical" > signature conflict? should they have to rebuild the package? Packages should not be accepted into any official repo, sid included, without some verification builds. A .deb should remain unchanged once it is accepted into any official repo (maybe experimental could be an exception, but not sid). I think that is essential. I see no reason for changing the .deb between sid and testing, except for perhaps how existing implementations are doing it. It is usually worth the work to do things right way, rather than the easy way. The build verification process needs to happen between the package upload and publishing to sid or security updates. Two builds is easy: the .deb that the uploader generates and the one the Debian process makes. That is probably enough. In Debian's case, it probably is too complicated to include multiple signatures. In that case, there should be only one canonical signature by dak once the build verification signature threshold has been passed. Then all of the other signatures could be added to .buildinfo or .changes or whatever other file. Another option is to do it like f-droid.org does. F-droid.org generates a APK signing key for each app, then manages the signing on a specialized signing server. Or another option is just requiring all the signers to be from the debian-keyring, rather than an exact match for previous signers. In any case, the .deb needs to remain unchanged. > You're entirely right that when fetching files via the web directly > (instead of an apt repository) or sneakernet, people tend to transfer > only the minimal set of possible files, and therefore having detached > signatures is a bad idea for adoption. > > But i don't think this addresses the concern raised above that specific > .deb files have constant size and contents, which is an assumption that > permeates the repository, mirror, and distribution mechanisms. > Rejecting that assumption means potentially breaking a lot of > infrastructure that currently works, as well as forcing incompatible > policy changes on archive operators. > > So i'd like to have this cake and eat it too, please :) > > Here's a proposal for chewing over: > > * define a new package format called .debs > > * foo_1.2-3_mipsel.debs is a tarball that contains at least three files: > > foo_1.2-3_mipsel.deb > foo_1.2-3_mipsel.buildinfo > foo_1.2-3_mipsel.buildinfo.0EE5BE979282D80B9F7540F1CCD2ED94D21739E9.asc > > (it can contain more than one .asc if it wants to include multiple > signatures) > > * if you invoke "dpkg -i foo_1.2-3_mipsel.debs", dpkg should unpack and > inspect the .debs, and the signatures, and refuse to install the .deb if > the signatures don't meet local policy. (i'm hand-waving here about > what "local policy" is, since i think that's a separate discussion) > > Now we can leave the current online archive distribution alone -- apt > works (modulo bugs) and archive operators can continue to function as > they currently do. But we tell users and upstream developers that if > they want to install packages via sneakernet or by downloading them > individually from the web that they really should be passing around > .debs files, and not .deb files. We could even modify dpkg to reject > installations of plain .deb files unless a package manager (which has > presumably already verified the package by other means) is do
[Reproducible-builds] put commit messages on a separate list?
I fear that all these commit emails will drown out the communication between humans. Could the commit messages could be on a separate list, like using one of these official lists: https://packages.qa.debian.org/common/index.html Checkout git.debian.org:/git/collab-maint/setup-repository for an example of how to send commit messages to packages.qa.debian.org. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > On 09/19/2014 06:07 AM, Elmar Stellnberger wrote: >>Isn`t there really any way to include the signatures in the header of >> the .deb files? >> Why not simply add multiple signature files in the control.tar.gz of a >> .deb just next >> to the md5sums which should in deed be a sha256sums (otherwise there is >> no way >> to establish a 'chain of trust'). That would not add any non-determinism >> because >> if it is done right then we can have all the signers in the package! > > If we succeed in creating reproducible builds (we're farther along > toward that goal than i had dared hope, it's exciting!) then one of the > nice opportunites we have is for other people or organizations to > corroborate the build after the package is first distributed. For > example, an upload to sid might have one signature (by the original > uploader), but maybe it waits to transition to testing until there are > corroborations from multiple builders. (Note: this is not a concrete > proposal or an expectation of exactly what will happen, just a thought > experiment) > > In this scenario, how do you propose to add those signatures into the > package? If we bundle them into the .deb, then the size and digest of > the .deb itself changes after it is first distributed. This seems like > it would violate all sorts of existing expectations -- i can't imagine > how many different tools and pieces of infrastructure expect that > foo_1.2-3_mipsel.deb should always have the same size and digest. There needs to be a canonical, immutable signature (which could be a set of individual signatures) on the final, official package for this to be a workable system. I like the idea of having a threshold of builder signatures before a package is accepted into the official repo. That makes a canonical signature easy to: it would just be a whatever signatures that triggered the inclusion, then perhaps also a dak signature. Android's APK signature is a good example of this. The Android system uses TOFU/POP on the set of signatures. At least one signature is required for an APK to be installed, but it can also be many signatures. Then whenever updating, the new APK's set of signatures must match the installed APK's set. This means that it is easy to install test versions of packages as long as it is not updating an installed package with a different signature. >>It would be far better than detaching the signatures from the package >> because >> the general use case where you need package signatures is the manual >> download >> of packages. Detached signatures are completely useless for such a use >> case! > > I don't think this is the case. People who download a .deb and verify > it could also download the associated .buildinfo file and whichever > signatures they'd like (or all of them) and verify the package that way. > > --dkg I strongly disagree. Adding a separate file for the signature turns it into something that only security-conscious techie people will do because it turns it into a conscious decision and action. Only a minority of techies actively think about verifying software, it is basically a non-existent practice for non-techies. The signature and it's verification must be transparent to actually work for the 99% of users who do not think about this stuff. Having the signature embedded in the file itself means that .deb files can be freely swapped via whatever available means, be it USB thumb drives, direct download, LibraryBox, the cloud, SparkleShare, email, IM, whatever. Then the signature can be transparently verified by dpkg on install, and the user can be entirely ignorant that the signature even exists unless there is a problem with it. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] concrete steps for improving apt downloading security and privacy
Daniel Kahn Gillmor wrote: > On 09/19/2014 12:34 AM, Paul Wise wrote: >> On Fri, Sep 19, 2014 at 9:30 AM, Hans-Christoph Steiner wrote: >> >>> Finally did this: >>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762153 >> >> Please note that you proposal to add signatures to .deb files will >> break reproducible builds because the hash of the .deb will differ >> depending on who signed it: >> >> https://wiki.debian.org/ReproducibleBuilds >> >> I think it would be far better to ship detached signatures in the >> archive since that allows for reproducible builds and also means there >> could be more than one signer (say one buildd, one Debian sponsor and >> one package maintainer). > > I agree with pabs on this. > > fwiw, i'm also hoping that we can ship at least one other signature for > the upstream tarball (where such a thing exists): > > https://bugs.debian.org/759478 > > We also had a discussion in the reproducible-builds BoF at DC14 about > how to deal with signatures on .buildinfo files, and came to the same > conclusion: that a .buildinfo file should have detached signatures, to > allow for multiple (corroborative) signers: > > https://wiki.debian.org/ReproducibleBuilds#A.buildinfo_signatures > > Note that a signature over a .buildinfo file should effectively cover > the digest of the built .deb files, which should creates a strong > cryptographic chain if you trust the hash function. > > Given that we would ultimately like one or more signed .buildinfo files > shipped in the archive, and that they represent a way to have an > builder's signature over a .deb, i think these make the idea of an > internally-signed .deb redundant. > > Thanks to everyone who is thinking about and working on improving the > cryptographic integrity of the archive! > > --dkg Having the signature in the .deb file does not necessarily break reproducibility and it does produce a signature that is vastly easier to use than a detached file because if you have the file, you have the signature (see my offline TAILS example). First there needs to be a source of a canonical embedded signature, then that signature just needs to be copied into the package on each build. Since everything else is exactly the same, copying the signature works. That canonical signature could be from dak, from the maintainer, or perhaps elsewhere. There are many widespread precedents for including the signature in the package itself: .jar, .apk, .rpm, Windows packages, etc. The code for an embedded signature is more complicated than a detached signature, so there is more risk to that approach (for example, Android's Master Key bug). Since apt already provides a detached signature, I think the risk in Debian is much lower and usability gain is much larger so its a worthwhile trade-off. .hc -- PGP fingerprint: 5E61 C878 0F86 295C E17D 8677 9F0F E587 374B BE81 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
