Re: source-only builds and .buildinfo
On Wed 2017-06-21 15:42:07 +0100, Ian Jackson wrote: > This is a very useful concept but I suggest you give it a new name. > "binaries-attested upload" perhaps ? I like the idea that we should name this thing, but i'd call it something like a "source-only upload with .buildinfo" or "source+buildinfo upload" instead. > To me "source-only upload" means that there were no binaries built, > and therefore no information about binaries included in the upload. i tend to think "source-only" in this phrase applies to "upload", meaning that the upload doesn't include binaries, and what i'm uploading doesn't include binaries. i acknowledge that it also includes some stuff that isn't actually sources, but this is true of normal "source-only" uploads too -- for example, such uploads include cryptographic signatures and selected elements of the changelogs, which are also not sources. ☺, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: Please review the draft for week 112's blog post
Chris Lamb wrote: > Please review the draft for week 112's blog post: [..] This has now been published at: https://reproducible.alioth.debian.org/blog/posts/112/ Many thanks all :) Regards, -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁Chris Lamb ⢿⡄⠘⠷⠚⠋ la...@debian.org / chris-lamb.co.uk ⠈⠳⣄ ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Daniel Kahn Gillmor writes ("Re: source-only builds and .buildinfo"):
> On Wed 2017-06-21 13:38:42 +0100, Ian Jackson wrote:
> > Certainly `dgit push' will not do anything to any .buildinfo you may
> > have. I think maybe that your use case should be supported by having
> > a version of dgit push which drops the .debs from the .changes, but
> > leaves the .buildinfo ? Is that how you construct these uploads now ?
>
> I really don't have to do anything manually. The standard
> dpkg-buildpackage toolchain does it for me if i pass
> --changes-option=-S -- it all works as i expect, and kudos to the dpkg
> developers for that :)
Then I think `dgit ... sbuild ...' (a binaryful build) followed by
`dgit --ch:-S push' (a binaryless upload) will probably do the same
thing.
Definitely in this case, dgit ought not to mess with the .buildinfo.
(Ie I think it will be included in the .changes, and dgit ought to
leave it there.)
> c) given this explicit set of build dependencies, here are the digests
> of the binary packages that were produced on my system.
>
> You say "verify my assertions about the .debs", i think you're talking
> about part (c), but there's nothing specifically to verify there. I'm
> saying to the world what *i* found when i built them. You want to tell
> me you found something different? fine! Now we have something to
> investigate. You found the same thing? Great, but that's a
> corroboration, not a verification.
Well, (c) is only useful if the build "is" reproducible. (That is,
"is reproducible in some plausible scenario".)
> But i don't think that we need to officially "close the loop" in any
> fancy (or strict) way to warrant shipping .buildinfo files from
> developers. The fancy console i propose above (or anything like it) can
> only be built and used across the archive once we have shipped the
> .buildinfo files. Unnecessarily stripping .buildinfo files that we know
> about only delays that process.
My comments here are more of an aside. I'm certainly not suggesting
that theis line of reasoning suggests any .buildinfos should be
stripped; merely that if I were you I would want to see about closing
this loop so because right now you are perhaps generating .buildinfos
which are going to be difficult to use this way in the future.
If some routine consumer of these .buildinfos comes into being, then
it would definitely be a good idea for dgit to gain convenient and
meaningful option(s) to generate such uploads. More convenient than
`--ch:-S' (which is using an escape hatch, and hence undesirable for
routine use).
However, `dgit push-source' is a different case. That is a command
where the dgit user asks dgit to upload their package source code to
Debian, but without doing any kind of binary package build at all.
(Probably the user has done some kind of pre-upload test to check that
the source does generate working binaries, but perhaps of a source
package with a different version in the changelog, or something.)
In that case, dpkg-buildpackage currently does still generate a
.buildinfo. That .buildinfo does not contain any information about
binary package builds - since there were no binary package builds.
Nor is the build-dependency information in the .buildinfo particularly
useful even for figuring out in what circumstances the uploader was
able to successfully run `debian/rules clean'. The experienced [d]git
user will probably be cleaning their working trees with git clean, not
rules clean. And, regardless, even if the uploader did run rules
clean, this has no bearing on the source package that gets uploaded,
since dgit verifies that the source package is identical to the
uploader's git tree.
Part of the confusion in this thread is, I think, due to the
overloading of the term "source-only upload" for your hybrid upload
which did _build_ binaries, and describes them in the .buildinfo, but
does not actually _ship_ them.
This is a very useful concept but I suggest you give it a new name.
"binaries-attested upload" perhaps ?
To me "source-only upload" means that there were no binaries built,
and therefore no information about binaries included in the upload.
Regards,
Ian.
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed 2017-06-21 13:38:42 +0100, Ian Jackson wrote: > This is an interesting use case which dgit should support. cool! > But I think this is not what dgit push-source should do. Sean's > proposed dgit push-source does not do any kind of binary package > build. I think this is correct. But this means there are no binaries > and nothing for the .buildinfo to talk about. I don't know anything about "dgit push-source", so i defer to you on that. > Do the "source-only uploads" that you are talking about mention the > hashes of these locally-built .debs in their .buildinfo, then ? yes. when building foo version 1.2-3, the .changes file mentions only: - foo_1.2-3.dsc - foo_1.2.orig.tar.bz2 - foo_1.2-3.debian.tar.bz2 - foo_1.2-3_amd64.buildinfo and the buildinfo file mentions: - foo_1.2-3.dsc - libfoo_1.2-3.deb - foo-tools_1.2-3.deb I *do not* upload any of the *.deb files to the archive. > Certainly `dgit push' will not do anything to any .buildinfo you may > have. I think maybe that your use case should be supported by having > a version of dgit push which drops the .debs from the .changes, but > leaves the .buildinfo ? Is that how you construct these uploads now ? I really don't have to do anything manually. The standard dpkg-buildpackage toolchain does it for me if i pass --changes-option=-S -- it all works as i expect, and kudos to the dpkg developers for that :) > (Also: is there anything right now that verifies your assertions about > the .debs? Not that the lack of such a thing would make the > .buildinfos useless, but my experience is that without closing that > loop it is likely that the arrangements for generating the .buildinfo > are wrong somehow in a way we haven't spotted.) In a standard upload of the type i'm describing i've asserted: a) I built version 1.2-3 on amd64, and it should be included in debian b) here are the digests of the source code (including debian packaging) c) given this explicit set of build dependencies, here are the digests of the binary packages that were produced on my system. You say "verify my assertions about the .debs", i think you're talking about part (c), but there's nothing specifically to verify there. I'm saying to the world what *i* found when i built them. You want to tell me you found something different? fine! Now we have something to investigate. You found the same thing? Great, but that's a corroboration, not a verification. I agree with you that it'd be nice in the future to "close the loop" by having infrastructure that monitors all of these developer-generated .buildinfo files, compares them to the buildd-generated .buildinfo files, and provides some sort of interface for easy reasoning about them. and such infrastructure could well show that something is wrong with how we're generating .buildinfo files; that's fine, we'd then modify how we generate buildinfo files going forward to correct it, if necessary. Imagine a fancy console that a debian developer could pull up which shows a list of binary packages they submitted which differ from the one being shipped by the archive, and which build-dependencies it noticed were different (or, just shows a green light if it's the case all of their current uploads have been corroboratively rebuilt). cool, eh? Or some future, stricter debian variant might even want to only allow a package to enter the archive if the binary packages created by the buildd of the submitted architecture match the binaries claimed by the submitting developer (i'm *not* proposing this for debian today. it could introduce hassle and delay because of the concerns about build-dep synchronization that Adrian raises, and we don't have the workflow for it smooth enough yet). But i don't think that we need to officially "close the loop" in any fancy (or strict) way to warrant shipping .buildinfo files from developers. The fancy console i propose above (or anything like it) can only be built and used across the archive once we have shipped the .buildinfo files. Unnecessarily stripping .buildinfo files that we know about only delays that process. Regards, --dkg signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On 2017-06-21, Ian Jackson wrote:
> Daniel Kahn Gillmor writes ("Re: source-only builds and .buildinfo"):
>> On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote:
>> > A .buildinfo file is not useful for a source-only upload which is
>> > veried to be identical to the intended source as present in the
>> > uploader's version control (eg, by the use of dgit).
>> >
>> > Therefore, dgit should not include .buildinfos in source-only uploads
>> > it performs. If dgit sees that a lower-layer tool like
>> > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit
>> > should strip it out of .changes.
>>
>> I often do source-only uploads which include the .buildinfo.
>>
>> I do source-only uploads because i don't want the binaries built on my
>> own personal infrastructure to reach the public. But i want to upload
>> the .buildinfo because i want to provide a corroboration of what i
>> *expect* the buildds to produce.
>
> This is an interesting use case which dgit should support.
Agreed!
> But I think this is not what dgit push-source should do. Sean's
> proposed dgit push-source does not do any kind of binary package
> build. I think this is correct. But this means there are no binaries
> and nothing for the .buildinfo to talk about.
Yes, this makes sense for the most part.
> Do the "source-only uploads" that you are talking about mention the
> hashes of these locally-built .debs in their .buildinfo, then ?
That's the goal, sure.
I've done this with all my recent source-only uploads, and then gone
back and verified that the buildd machines produced (in most cases), the
same hashes for the .deb files.
For example, this references the buildinfo of simple-cdd 0.6.5 I
uploaded with a source-only changes file in:
https://buildinfo.debian.net/30f7000b0025b570c7ae2202fc6fd79e4ca27798/simple-cdd_0.6.5_all
And this is a buildinfo produced over a month later on the reproducible
builds build network, on a different architecture (i386), with a
different build environment, that produced the same hashes:
https://buildinfo.debian.net/1d300b71445ac7d756e93546a7e6b36d3c1882c7/simple-cdd_0.6.5_all
And you can check the .buildinfo in the build logs on the buildd
produced the same sha1 hashes:
https://buildd.debian.org/status/fetch.php?pkg=simple-cdd&arch=all&ver=0.6.5&stamp=1494884527&raw=0
And then you can compare the hashes of simple-cdd packages in the
archive are the same hashes listed.
Given that at least three machines, of differing architecture, with over
a month between the packages in the build toolchain, produced the same
binary packages... I have *some* confidence that this package is
reproducible.
It's not the most complicated package, but it demonstrates that it is
now possible, for a reasonable portion of the archive, to at least
manually verify many of the builds. Some of this could be automated...
> Certainly `dgit push' will not do anything to any .buildinfo you may
> have. I think maybe that your use case should be supported by having
> a version of dgit push which drops the .debs from the .changes, but
> leaves the .buildinfo ? Is that how you construct these uploads now ?
I use sbuild's --source-only-changes option, which creates two .changes
files, one with the debs (ARCH.changes), and one
without(source.changes). In both cases, the .buildinfo referenced in
.changes includes hashes of the .deb files.
> (Also: is there anything right now that verifies your assertions about
> the .debs? Not that the lack of such a thing would make the
> .buildinfos useless, but my experience is that without closing that
> loop it is likely that the arrangements for generating the .buildinfo
> are wrong somehow in a way we haven't spotted.)
There's nothing corroborating the results of .deb files in the archive
against tests.reproducible-builds.org build results, but that does
rebuild all packages in the archive with permutations of the build
environment, and logs when they aren't reproducible.
The archive is keeping the .buildinfo files uploaded with packages,
though they aren't, to my knowledge, exposed yet. But it would allow for
retroactive verification of said packages once the .buildinfo files are
available. A few relevent bugs on ftp.debian.org regarding this:
https://bugs.debian.org/763822
https://bugs.debian.org/862073
https://bugs.debian.org/862538
https://bugs.debian.org/863470
live well,
vagrant
signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed 2017-06-21 14:16:00 +0300, Adrian Bunk wrote: > On Wed, Jun 21, 2017 at 09:30:14AM +, Holger Levsen wrote: >> Hi, >> >> trigger warning: nitpicking. >> >> On Wed, Jun 21, 2017 at 11:34:17AM +0300, Adrian Bunk wrote: >> > > I do source-only uploads because i don't want the binaries built on my >> > > own personal infrastructure to reach the public. But i want to upload >> > > the .buildinfo because i want to provide a corroboration of what i >> > > *expect* the buildds to produce. >> > If you expect that, then your expectation is incorrect. >> >> I actually think that dkg's expectation is right, "just" that reality is >> wrong. >> >> The design of the Debian buildd network is from times when machines were much >> less powerful than what we have today and it shows. >> >> I'd rather have deterministic builds than the current unpredictable mess. > > I understand what you want, but using buildinfo is not a good idea here. > > Based on how many broken binaries get uploaded from developers, > the environment of the person uploading the sources is not a good > basis for determining what package versions to install when building > on the buildds. lest there be any misunderstanding: i am *not* suggesting that i want the build daemons to select their packages based on what's in my .buildinfo. Ximin's interpretation of my intent is the correct one: i want to see whether we manage to reproduce the same output. if the binary package outputs differ, and the installed build-deps differ, fine. that's data that someone tracking how things are built can use in a future analysis. if the binar package outputs do *not* differ, and the build-deps differ, that's also interesting information. my goal here isn't to use the build daemons as the r-b infrastructure -- we've already got the r-b infrastructure for that. :) But i'm happy to be able to see some corroborative (or anti-corroborative) .buildinfos published so that people who want to analyze them can do so. --dkg PS I fully agree that the right outcome for debian overall is to not allow binary uploads from anyone, unless they're granted special dispensation (e.g. toolchain porters), but that's getting far afield from this thread. signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Daniel Kahn Gillmor writes ("Re: source-only builds and .buildinfo"):
> On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote:
> > A .buildinfo file is not useful for a source-only upload which is
> > veried to be identical to the intended source as present in the
> > uploader's version control (eg, by the use of dgit).
> >
> > Therefore, dgit should not include .buildinfos in source-only uploads
> > it performs. If dgit sees that a lower-layer tool like
> > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit
> > should strip it out of .changes.
>
> I often do source-only uploads which include the .buildinfo.
>
> I do source-only uploads because i don't want the binaries built on my
> own personal infrastructure to reach the public. But i want to upload
> the .buildinfo because i want to provide a corroboration of what i
> *expect* the buildds to produce.
This is an interesting use case which dgit should support.
But I think this is not what dgit push-source should do. Sean's
proposed dgit push-source does not do any kind of binary package
build. I think this is correct. But this means there are no binaries
and nothing for the .buildinfo to talk about.
Do the "source-only uploads" that you are talking about mention the
hashes of these locally-built .debs in their .buildinfo, then ?
Certainly `dgit push' will not do anything to any .buildinfo you may
have. I think maybe that your use case should be supported by having
a version of dgit push which drops the .debs from the .changes, but
leaves the .buildinfo ? Is that how you construct these uploads now ?
(Also: is there anything right now that verifies your assertions about
the .debs? Not that the lack of such a thing would make the
.buildinfos useless, but my experience is that without closing that
loop it is likely that the arrangements for generating the .buildinfo
are wrong somehow in a way we haven't spotted.)
> why wouldn't dgit take the same approach? stripping the .buildinfo from
> the .changes seems like a wasted shot at a potential corroboration.
> or am i misunderstanding the question here?
I hope you're misunderstanding. I'm open to being convinced I'm
wrong...
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed, Jun 21, 2017 at 02:16:00PM +0300, Adrian Bunk wrote: > Based on how many broken binaries get uploaded from developers, we should disallow binary uploads for everybody for all packages by default. those porters who need it, should get that enabled for those packages where they need it, when they need. -- cheers, Holger signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed, Jun 21, 2017 at 09:30:14AM +, Holger Levsen wrote: > Hi, > > trigger warning: nitpicking. > > On Wed, Jun 21, 2017 at 11:34:17AM +0300, Adrian Bunk wrote: > > > I do source-only uploads because i don't want the binaries built on my > > > own personal infrastructure to reach the public. But i want to upload > > > the .buildinfo because i want to provide a corroboration of what i > > > *expect* the buildds to produce. > > If you expect that, then your expectation is incorrect. > > I actually think that dkg's expectation is right, "just" that reality is > wrong. > > The design of the Debian buildd network is from times when machines were much > less powerful than what we have today and it shows. > > I'd rather have deterministic builds than the current unpredictable mess. I understand what you want, but using buildinfo is not a good idea here. Based on how many broken binaries get uploaded from developers, the environment of the person uploading the sources is not a good basis for determining what package versions to install when building on the buildds. > cheers, > Holger cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed, Jun 21, 2017 at 10:09:00AM +, Ximin Luo wrote: > Adrian Bunk: > > On Wed, Jun 21, 2017 at 09:28:00AM +, Ximin Luo wrote: > >> Adrian Bunk: > >>> On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote: > Hi Ian-- > > On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: > > A .buildinfo file is not useful for a source-only upload which is > > veried to be identical to the intended source as present in the > > uploader's version control (eg, by the use of dgit). > > > > Therefore, dgit should not include .buildinfos in source-only uploads > > it performs. If dgit sees that a lower-layer tool like > > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit > > should strip it out of .changes. > > I often do source-only uploads which include the .buildinfo. > > I do source-only uploads because i don't want the binaries built on my > own personal infrastructure to reach the public. But i want to upload > the .buildinfo because i want to provide a corroboration of what i > *expect* the buildds to produce. > ... > >>> > >>> If you expect that, then your expectation is incorrect. > >>> > >>> If you upload a package right now, chances are the buildds will use both > >>> older versions of some packages [1] and more recent versions of some > >>> other packages [2] than what you used. > >>> > >> > >> I think what dkg means here (and what we the R-B team has wanted for ages > >> and is working towards), is not that the buildds use the *versioned > >> dependencies* listed in the buildinfo, but produce the same *output > >> hashes* as what's in the buildinfo. > >> > >> The point being specifically that the dependencies used could change, but > >> if the output remains constant, we're more assured that the build was done > >> properly and reproducibly. > > > > How is that supposed to work when the compiler is not exactly identical? > > > > As an example, gcc-6 6.3.0-18 and gcc-6 6.3.0-19 will likely produce > > different output for every non-trivial piece of software. > > > > The reason is that every new gcc upload usually contains whatever > > bugfixes are on the upstream branch. > > > > It would depend on the situation which dependencies should be "irrelevant" > towards the final output, right. If the dependencies are different and the > buildinfo is different, it does not necessarily mean there is a problem, the > upload does not need to be rejected. But it's a signal that other people > (including the uploader) might want to re-try the build with the newer > dependencies. > > OTOH if the outputs match, we get more certainty, which is a good thing. >... "more certainty" on what exactly? "signal that other people might want to" is quite vague, what do you want to prove and how exactly should people spend time proving it? In the best case [1] we would know that the buildd on the one architecture that happens to be used by the person doing the source upload produced the same binaries. Once you start verifying that all binaries in the archive were built from the sources in the archive, this will automatically be covered. > X cu Adrian [1] excluding the binary-all special case -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Запис на телефонни разговори! Научи повече >>
02 874 00 80 Прослушайте всички важни разговори Телефонна централа от ново поколение с още много възможности ЗАЯВЕТЕ ДЕМО СЕГА! ПРОБЛЕМИ: Липса на записи Пропускане на детайли Забравяне на информация Липса на контрол Поддържани устройства: РЕШЕНИЯ: Запис на разговори Възможност за прослушване Пълна информация Контрол Поддържани устройства: Подобрете качеството на обслужване Вече няма да се притеснявате дали сте въвели коректно поръчка за клиент, винаги може да прослушате разговор. Без ограничение на работното място Спокойствие по време на разговор с клиент, където и да се намирате. Вече не е нужно да запомняте всички детайли от разговора. Подобрете вашите лични търговски умения или на вашия търговски отдел Всеки търговец и мениджър има на разположение записите на всички разговори. Телефонна централа от ново поколение – крачка пред останалите! Ако желаете да разберете повече за възможностите, екипа на Callflow е на разположение да ви съдейства, за да направите крачка към ефективни бизнес процеси. ЗАЯВЕТЕ ДЕМО СЕГА! 02 874 00 80 Съгласно Закона за Електронната Търговия ви информираме, че това може да е непоискано търговско съобщение.Вие може да го приемете или отхвърлите. Ние може да сме получили вашия имейл от вас, от ваш партньор или от публичното пространство. Ако не желаете да получавате повече информацияот "Callflow", моля натиснете ОТПИСВАНЕ и ние ще ви премахнем от нашите списъци!Ако сме Ви обеспокоили, моля да ни извините. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Adrian Bunk: > On Wed, Jun 21, 2017 at 09:28:00AM +, Ximin Luo wrote: >> Adrian Bunk: >>> On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote: Hi Ian-- On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: > A .buildinfo file is not useful for a source-only upload which is > veried to be identical to the intended source as present in the > uploader's version control (eg, by the use of dgit). > > Therefore, dgit should not include .buildinfos in source-only uploads > it performs. If dgit sees that a lower-layer tool like > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit > should strip it out of .changes. I often do source-only uploads which include the .buildinfo. I do source-only uploads because i don't want the binaries built on my own personal infrastructure to reach the public. But i want to upload the .buildinfo because i want to provide a corroboration of what i *expect* the buildds to produce. ... >>> >>> If you expect that, then your expectation is incorrect. >>> >>> If you upload a package right now, chances are the buildds will use both >>> older versions of some packages [1] and more recent versions of some >>> other packages [2] than what you used. >>> >> >> I think what dkg means here (and what we the R-B team has wanted for ages >> and is working towards), is not that the buildds use the *versioned >> dependencies* listed in the buildinfo, but produce the same *output hashes* >> as what's in the buildinfo. >> >> The point being specifically that the dependencies used could change, but if >> the output remains constant, we're more assured that the build was done >> properly and reproducibly. > > How is that supposed to work when the compiler is not exactly identical? > > As an example, gcc-6 6.3.0-18 and gcc-6 6.3.0-19 will likely produce > different output for every non-trivial piece of software. > > The reason is that every new gcc upload usually contains whatever > bugfixes are on the upstream branch. > It would depend on the situation which dependencies should be "irrelevant" towards the final output, right. If the dependencies are different and the buildinfo is different, it does not necessarily mean there is a problem, the upload does not need to be rejected. But it's a signal that other people (including the uploader) might want to re-try the build with the newer dependencies. OTOH if the outputs match, we get more certainty, which is a good thing. We also need to get some real data on this, it could be that a change from -18 to -19 would only affect a small number of packages, and most other ones would actually be compiled identically. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Wed, Jun 21, 2017 at 09:28:00AM +, Ximin Luo wrote: > Adrian Bunk: > > On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote: > >> Hi Ian-- > >> > >> On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: > >>> A .buildinfo file is not useful for a source-only upload which is > >>> veried to be identical to the intended source as present in the > >>> uploader's version control (eg, by the use of dgit). > >>> > >>> Therefore, dgit should not include .buildinfos in source-only uploads > >>> it performs. If dgit sees that a lower-layer tool like > >>> dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit > >>> should strip it out of .changes. > >> > >> I often do source-only uploads which include the .buildinfo. > >> > >> I do source-only uploads because i don't want the binaries built on my > >> own personal infrastructure to reach the public. But i want to upload > >> the .buildinfo because i want to provide a corroboration of what i > >> *expect* the buildds to produce. > >> ... > > > > If you expect that, then your expectation is incorrect. > > > > If you upload a package right now, chances are the buildds will use both > > older versions of some packages [1] and more recent versions of some > > other packages [2] than what you used. > > > > I think what dkg means here (and what we the R-B team has wanted for ages and > is working towards), is not that the buildds use the *versioned dependencies* > listed in the buildinfo, but produce the same *output hashes* as what's in > the buildinfo. > > The point being specifically that the dependencies used could change, but if > the output remains constant, we're more assured that the build was done > properly and reproducibly. How is that supposed to work when the compiler is not exactly identical? As an example, gcc-6 6.3.0-18 and gcc-6 6.3.0-19 will likely produce different output for every non-trivial piece of software. The reason is that every new gcc upload usually contains whatever bugfixes are on the upstream branch. > X cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Hi, trigger warning: nitpicking. On Wed, Jun 21, 2017 at 11:34:17AM +0300, Adrian Bunk wrote: > > I do source-only uploads because i don't want the binaries built on my > > own personal infrastructure to reach the public. But i want to upload > > the .buildinfo because i want to provide a corroboration of what i > > *expect* the buildds to produce. > If you expect that, then your expectation is incorrect. I actually think that dkg's expectation is right, "just" that reality is wrong. The design of the Debian buildd network is from times when machines were much less powerful than what we have today and it shows. I'd rather have deterministic builds than the current unpredictable mess. -- cheers, Holger signature.asc Description: Digital signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
package uploaded to our repo
gcc-6_6.3.0-19.0~reproducible1.dsc has just been uploaded to https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
Adrian Bunk: > On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote: >> Hi Ian-- >> >> On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: >>> A .buildinfo file is not useful for a source-only upload which is >>> veried to be identical to the intended source as present in the >>> uploader's version control (eg, by the use of dgit). >>> >>> Therefore, dgit should not include .buildinfos in source-only uploads >>> it performs. If dgit sees that a lower-layer tool like >>> dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit >>> should strip it out of .changes. >> >> I often do source-only uploads which include the .buildinfo. >> >> I do source-only uploads because i don't want the binaries built on my >> own personal infrastructure to reach the public. But i want to upload >> the .buildinfo because i want to provide a corroboration of what i >> *expect* the buildds to produce. >> ... > > If you expect that, then your expectation is incorrect. > > If you upload a package right now, chances are the buildds will use both > older versions of some packages [1] and more recent versions of some > other packages [2] than what you used. > I think what dkg means here (and what we the R-B team has wanted for ages and is working towards), is not that the buildds use the *versioned dependencies* listed in the buildinfo, but produce the same *output hashes* as what's in the buildinfo. The point being specifically that the dependencies used could change, but if the output remains constant, we're more assured that the build was done properly and reproducibly. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: source-only builds and .buildinfo
On Tue, Jun 20, 2017 at 02:47:20PM -0400, Daniel Kahn Gillmor wrote: > Hi Ian-- > > On Tue 2017-06-20 18:10:49 +0100, Ian Jackson wrote: > > A .buildinfo file is not useful for a source-only upload which is > > veried to be identical to the intended source as present in the > > uploader's version control (eg, by the use of dgit). > > > > Therefore, dgit should not include .buildinfos in source-only uploads > > it performs. If dgit sees that a lower-layer tool like > > dpkg-buildpackage provided a .buildinfo for a source-only upload, dgit > > should strip it out of .changes. > > I often do source-only uploads which include the .buildinfo. > > I do source-only uploads because i don't want the binaries built on my > own personal infrastructure to reach the public. But i want to upload > the .buildinfo because i want to provide a corroboration of what i > *expect* the buildds to produce. >... If you expect that, then your expectation is incorrect. If you upload a package right now, chances are the buildds will use both older versions of some packages [1] and more recent versions of some other packages [2] than what you used. > --dkg cu Adrian [1] buildd chroots are regenerated twice per week and not updated prior to each build [2] some packages might already have been updated compared to what you used -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
