Bootstrappable Debian - a decision is needed, patches exist
Hi, This email is a follow up on the thread started January 2013 [1]. In summary: it seems that the ability to bootstrap Debian from scratch and the requirement to extend the Build-Depends syntax meet general agreement. What is yet to be decided is the concrete format for the Build-Depends syntax extension. The first proposals suggested a syntax which looked like Build-Depends: foo [amd64] Which would indicate that the build dependency "foo" would not apply if the build profile called "stage1" is activated. It was critisized [2] that this syntax wastes a meta character and thus prohibits future extensions of the Build-Depends syntax. Therefore the second proposal (finalised at debconf13) looked like this: Build-Depends: foo [amd64] [!profile.stage1] The rectangular brackets are reused and a prefixed namespace is used to indicate that "stage1" is a build profile name. We hoped this would be the final spec, given the previous discussion, but those brackets also got some pushback [3] and thus the third version was born: Build-Depends: foo [amd64] We wrote down the last two options in form of a spec on the Debian wiki [11]. Patches for dpkg, python-debian, apt and sbuild implementing the original format have existed for years [4]. Patches for the new formats have existed for some time as well [5]. They are surely not perfect but we would like to get them into a state in which they can be integrated into dpkg. But for that we need some feedback from the dpkg devs as well as a final decision of the Debian community about which syntax to choose. We are writing to d-devel this time because the thread on d-dpkg [6,7] has been dormant for a month once again. Maybe bringing this issue to a wider audience will help make a decision on this problem. The results from two years of GSoC [8,9] as well as the year long efforts of others [10] cannot bear any fruit without this issue finally being taken care of. Thank you! josch & wookey [1] http://lists.debian.org/20130115181840.GA16417@hoothoot [2] http://lists.debian.org/20726.45081.38709.233...@chiark.greenend.org.uk [3] http://lists.debian.org/20130816121504.gb20...@gaara.hadrons.org [4] http://people.debian.org/~wookey/bootstrap/patches/profiles/tools/ [5] http://lists.debian.org/20130917103117.2726.40546@hoothoot [6] http://lists.debian.org/20130419194252.17205.76995@hoothoot [7] http://lists.debian.org/debian-dpkg/2013/08/msg00019.html [8] http://www.alkmim.eti.br/~alkmim/gitrepo/autobootstrap.git [9] https://gitorious.org/debian-bootstrap/botch [10] http://people.debian.org/~wookey/bootstrap [11] https://wiki.debian.org/BuildProfileSpec -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131015060337.7934.42627@hoothoot
Re: Bootstrappable Debian - a decision is needed, patches exist
On Tue, Oct 15, 2013 at 2:03 PM, Johannes Schauer wrote: > Hi, > > This email is a follow up on the thread started January 2013 [1]. In summary: > it seems that the ability to bootstrap Debian from scratch and the requirement > to extend the Build-Depends syntax meet general agreement. > > What is yet to be decided is the concrete format for the Build-Depends syntax > extension. The first proposals suggested a syntax which looked like > > Build-Depends: foo [amd64] I'd prefer Build-Depends-Stage1 if possible. When bootstrap, dpkg only ask for these build-depends while for normal build, dpkg should merge Build-Depends-Stage1 and Build-Depends. > > Which would indicate that the build dependency "foo" would not apply if the > build profile called "stage1" is activated. It was critisized [2] that this > syntax wastes a meta character and thus prohibits future extensions of the > Build-Depends syntax. Therefore the second proposal (finalised at debconf13) > looked like this: > > Build-Depends: foo [amd64] [!profile.stage1] > > The rectangular brackets are reused and a prefixed namespace is used to > indicate that "stage1" is a build profile name. We hoped this would be the > final spec, given the previous discussion, but those brackets also got some > pushback [3] and thus the third version was born: > > Build-Depends: foo [amd64] > > We wrote down the last two options in form of a spec on the Debian wiki [11]. > > Patches for dpkg, python-debian, apt and sbuild implementing the original > format have existed for years [4]. Patches for the new formats have existed > for > some time as well [5]. They are surely not perfect but we would like to get > them into a state in which they can be integrated into dpkg. But for that we > need some feedback from the dpkg devs as well as a final decision of the > Debian > community about which syntax to choose. We are writing to d-devel this time > because the thread on d-dpkg [6,7] has been dormant for a month once again. > Maybe bringing this issue to a wider audience will help make a decision on > this > problem. The results from two years of GSoC [8,9] as well as the year long > efforts of others [10] cannot bear any fruit without this issue finally being > taken care of. > > Thank you! > > josch & wookey > > [1] http://lists.debian.org/20130115181840.GA16417@hoothoot > [2] http://lists.debian.org/20726.45081.38709.233...@chiark.greenend.org.uk > [3] http://lists.debian.org/20130816121504.gb20...@gaara.hadrons.org > [4] http://people.debian.org/~wookey/bootstrap/patches/profiles/tools/ > [5] http://lists.debian.org/20130917103117.2726.40546@hoothoot > [6] http://lists.debian.org/20130419194252.17205.76995@hoothoot > [7] http://lists.debian.org/debian-dpkg/2013/08/msg00019.html > [8] http://www.alkmim.eti.br/~alkmim/gitrepo/autobootstrap.git > [9] https://gitorious.org/debian-bootstrap/botch > [10] http://people.debian.org/~wookey/bootstrap > [11] https://wiki.debian.org/BuildProfileSpec > > > -- > To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org > with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org > Archive: http://lists.debian.org/20131015060337.7934.42627@hoothoot > -- YunQiang Su -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAKcpw6VGjrC=++ka2cjq8bp6r8w8y4odviddtfr2nunsjlk...@mail.gmail.com
Re: Bootstrappable Debian - a decision is needed, patches exist
Hi, Quoting YunQiang Su (2013-10-15 08:08:52) > On Tue, Oct 15, 2013 at 2:03 PM, Johannes Schauer wrote: > > What is yet to be decided is the concrete format for the Build-Depends > > syntax extension. The first proposals suggested a syntax which looked like > > > > Build-Depends: foo [amd64] > I'd prefer Build-Depends-Stage1 if possible. > When bootstrap, dpkg only ask for these build-depends while for normal build, > dpkg should merge Build-Depends-Stage1 and Build-Depends. this approach does not work because it does not allow to specify additional dependencies for when a source package is compiled with a certain profile activated. The Build-Depends-Stage1 field name was used in earlier proposals with a different meaning. Instead of making the final build dependencies the union of Build-Depends and Build-Depends-Stage1 as you proposed, Build-Depends-Stage1 contained all dependencies necessary for the build profile "stage1". When doing a normal build, only the Build-Depends would be used. When building with the build profile "stage1" activated only the Build-Depends-Stage1 would be used. This solves the problem I explained above which your solution has. This Build-Depends-Stage1 format was abolished in favor of the syntax additions explained in my last email because of the following reasons: - it requires variable field names to match all possible profiles - it requires a near duplication of the Build-Depends field which is highly prone to bitrot - it does not work with multiple profiles activated at the same time For a full history of the development of this see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661538 cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131015063400.7934.32962@hoothoot
Re: Bootstrappable Debian - a decision is needed, patches exist
On Tue, 2013-10-15 at 08:03 +0200, Johannes Schauer wrote: [...] > But for that we > need some feedback from the dpkg devs as well as a final decision of the > Debian > community about which syntax to choose. [...] You, and the dpkg maintainers, are the ones that understand the consequences of the different syntaxes. Don't invite bikeshedding. If the dpkg maintainers refuse to make a decision then you should go to the technical committee. Hopefully that will not be necessary. Ben. -- Ben Hutchings Reality is just a crutch for people who can't handle science fiction. signature.asc Description: This is a digitally signed message part
Re: Bootstrappable Debian - a decision is needed, patches exist
Hi Johannes, On Tue, Oct 15, 2013 at 08:34:00AM +0200, Johannes Schauer wrote: > Quoting YunQiang Su (2013-10-15 08:08:52) > > On Tue, Oct 15, 2013 at 2:03 PM, Johannes Schauer > > wrote: > > > What is yet to be decided is the concrete format for the Build-Depends > > > syntax extension. The first proposals suggested a syntax which looked like > > > > > > Build-Depends: foo [amd64] > > I'd prefer Build-Depends-Stage1 if possible. > > When bootstrap, dpkg only ask for these build-depends while for normal > > build, > > dpkg should merge Build-Depends-Stage1 and Build-Depends. > this approach does not work because it does not allow to specify additional > dependencies for when a source package is compiled with a certain profile > activated. > The Build-Depends-Stage1 field name was used in earlier proposals with a > different meaning. Instead of making the final build dependencies the union of > Build-Depends and Build-Depends-Stage1 as you proposed, Build-Depends-Stage1 > contained all dependencies necessary for the build profile "stage1". When > doing > a normal build, only the Build-Depends would be used. When building with the > build profile "stage1" activated only the Build-Depends-Stage1 would be used. > This solves the problem I explained above which your solution has. > This Build-Depends-Stage1 format was abolished in favor of the syntax > additions explained in my last email because of the following reasons: > - it requires variable field names to match all possible profiles > - it requires a near duplication of the Build-Depends field which is highly >prone to bitrot > - it does not work with multiple profiles activated at the same time > For a full history of the development of this see > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661538 My recollection is that the "abolishing" of the Build-Depends-Stage1 field was done by the same dpkg maintainer who you say is now not giving you feedback. It's elegant that a general-purpose syntax has been proposed, but elegance isn't worth anything if it doesn't result in shipping code. Being able to automatically bootstrap the Debian archive is self-evidently useful to the project. The other uses proposed for profiles are all corner cases that should be kept far, far away from the archive. I don't think the implementation of automated bootstrapping should be allowed to block on such pie-in-the-sky plans for profiles. My understanding is that all build-dependency loops in the archive can be broken with a single additional stage (stage1), so only one added profile and one added build-dependency field would be required. Yes, it could bitrot, but it's better than not having any of the data in the source packages at all. And if someone really finds this inelegant and insists that we should extend the syntax of the Build-Depends field, let them step up and make that happen instead of pointing at grandiose plans they're not making any effort to implement. A Build-Depends-Stage1 field requires no support from dpkg in the archive to implement. Cheers, -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: Bootstrappable Debian - a decision is needed, patches exist
Hi Steve, Quoting Steve Langasek (2013-10-20 05:46:15) > My recollection is that the "abolishing" of the Build-Depends-Stage1 field > was done by the same dpkg maintainer who you say is now not giving you > feedback. Correct. > It's elegant that a general-purpose syntax has been proposed, but elegance > isn't worth anything if it doesn't result in shipping code. > > Being able to automatically bootstrap the Debian archive is self-evidently > useful to the project. The other uses proposed for profiles are all corner > cases that should be kept far, far away from the archive. I don't think the > implementation of automated bootstrapping should be allowed to block on such > pie-in-the-sky plans for profiles. > > My understanding is that all build-dependency loops in the archive can be > broken with a single additional stage (stage1), so only one added profile > and one added build-dependency field would be required. Yes, it could > bitrot, but it's better than not having any of the data in the source > packages at all. I agree with you and from my talking to wookey, he probably agrees too, that *any* solution is fine as long as it's accepted by dpkg devs and allows us to finally make Debian bootstrappable. You nicely summarized the situation and I dont think I have much more to add. Yes, falling back to the Build-Depends-Stage1 proposal (for which patches also existed for over a year, see bug#661538) would also fully satisfy our needs for now until something fancier is needed. In addition, my analysis from last year showed that probably below 70 source packages have to be modified to make everything bootstrappable. If that is the case, then any future switch to something different might still be managable. > And if someone really finds this inelegant and insists that we should extend > the syntax of the Build-Depends field, let them step up and make that happen > instead of pointing at grandiose plans they're not making any effort to > implement. A Build-Depends-Stage1 field requires no support from dpkg in the > archive to implement. I would love if that person did. cheers, josch -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131020064754.10620.38358@hoothoot
Re: Bootstrappable Debian - a decision is needed, patches exist
+++ Steve Langasek [2013-10-19 20:46 -0700]: > Hi Johannes, > > My understanding is that all build-dependency loops in the archive can be > broken with a single additional stage (stage1), so only one added profile > and one added build-dependency field would be required. No, that's wrong. We need stage2 for at least toolchain bootstrapping. And practical usage has shown that distinguishing between bootstrap, cross and test deps is genuinely useful. Doko has also expressed this opinion, and in is one of the few people that has used this seriously. So I really wouldn't want to go back to the simple 'Build-Depends-Stage1: ' implementation. I agree with Guillem on this. > Yes, it could > bitrot, but it's better than not having any of the data in the source > packages at all. And if someone really finds this inelegant and insists > that we should extend the syntax of the Build-Depends field, let them step > up and make that happen instead of pointing at grandiose plans they're not > making any effort to implement. A Build-Depends-Stage1 field requires no > support from dpkg in the archive to implement. Right, but we have now written the code for the better scheme, so the only issue is actually putting it in dpkg, which applied just the same to Build-Depends-Stage1. That original scheme had an advantage when nothing better had been done or proposed. Now it has none. Guillem has now put this back on his list and promised to put somethig in soon, so I hope we'll see something which is both reasonably flexible and implemented very soon. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131023180136.gl7...@stoneboat.aleph1.co.uk
Re: Bootstrappable Debian - a decision is needed, patches exist
On Wed, Oct 23, 2013 at 07:01:36PM +0100, Wookey wrote: > +++ Steve Langasek [2013-10-19 20:46 -0700]: > > Hi Johannes, > > My understanding is that all build-dependency loops in the archive can be > > broken with a single additional stage (stage1), so only one added profile > > and one added build-dependency field would be required. > No, that's wrong. We need stage2 for at least toolchain bootstrapping. You need three distinct stages of package builds for this? AIUI the first stage of toolchain bootstrapping is always done outside the package build. > And practical usage has shown that distinguishing between bootstrap, cross > and test deps is genuinely useful. Doko has also expressed this opinion, > and in is one of the few people that has used this seriously. So I really > wouldn't want to go back to the simple 'Build-Depends-Stage1: ' > implementation. I agree with Guillem on this. Distinguishing between test deps and build-deps is purely an optimization, it doesn't enable any fundamentally new capabilities. Cross-deps don't seem to be covered by the current proposal AIUI. I agree that we're sorely missing a way to specify toolchain build-dependencies that need to change depending on whether you're doing a native or cross build; from my perspective this is even more important than bootstrapping, since you bootstrap a port once but run into packages with cross-toolchain-dependencies on an ongoing basis. But while profiles let us dodge unsatisfiable build-deps on things like , they don't provide the semantics to let us pull in in its place. Is anybody working on that problem? I think Colin's suggestion was that we might be better off having logic in apt's build-dep resolver to magically map build-deps in certain cases, but I don't know that this has been written down as a fleshed-out proposal anywhere yet. > > Yes, it could > > bitrot, but it's better than not having any of the data in the source > > packages at all. And if someone really finds this inelegant and insists > > that we should extend the syntax of the Build-Depends field, let them step > > up and make that happen instead of pointing at grandiose plans they're not > > making any effort to implement. A Build-Depends-Stage1 field requires no > > support from dpkg in the archive to implement. > Right, but we have now written the code for the better scheme, so the only > issue is actually putting it in dpkg, which applied just the same to > Build-Depends-Stage1. That original scheme had an advantage when nothing > better had been done or proposed. Now it has none. > Guillem has now put this back on his list and promised to put somethig in > soon, so I hope we'll see something which is both reasonably flexible and > implemented very soon. Even if this were implemented in dpkg today, you still can't merge this bootstrapping information into the Build-Depends fields of the packages in the archive until the autobuilders can cope with it. That means not only does dpkg have to be able to parse it, so do apt, wanna-build, and (IIRC) sbuild. When will that be done? The Build-Depends-Stage1 field requires no changes to the syntax of existing fields and no changes to the archive infrastructure. It could be added to packages immediately (or, you know, a year and a half ago). Build profiles may be a better overall design, but this design doesn't buy us very much in practical terms AFAICS. It just introduces more delays. Don't let the perfect be the enemy of the good. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: Bootstrappable Debian - a decision is needed, patches exist
On Wed, 23 Oct 2013 12:05:30 -0700, Steve Langasek wrote: > > And practical usage has shown that distinguishing between bootstrap, cross > > and test deps is genuinely useful. Doko has also expressed this opinion, > > and in is one of the few people that has used this seriously. So I really > > wouldn't want to go back to the simple 'Build-Depends-Stage1: ' > > implementation. I agree with Guillem on this. > Distinguishing between test deps and build-deps is purely an optimization, > it doesn't enable any fundamentally new capabilities. Slightly different topic but: Having build and test requirements separated would also help for autopkgtest, cf. eg. #720458. Cheers, gregor -- .''`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06 : :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/ `. `' Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe `- signature.asc Description: Digital signature
