Re: [arch-general] Question about automated builder
On Thu, Feb 17, 2011 at 06:25, Thomas S Hatch wrote: [...] > Heh, I like OCaml, but I chose Python for a few reasons, one was so that > more people could hack on it, I want it to be very modular, so for instance > it will be possible to make a variety of modules to watch different SCM > managers, and it wil be possible to write builder daemons in any language. > This will make it easy for me to port the system to a faster language if we > see the need. > > Also, python's libs are much more mature, and while I like OCaml a lot, I > have written hundreds of thousands of lines of production python, and far > less OCaml. > > I have a few mechanisms to make for package feeding, so for instance, it > will watch scm repos for changes and build PKGBUILDs found therein, but I > also plan to make an interface that can be used to just feed it source > packages. So I will ask for some people to send it source packages to test > once I get it going. > > Also, I will be asking people to try and break it with PKGBUILDS as well, to > see if PKGBUILDS can be made that can break out of the build root etc. All very good reasons indeed :-) /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus
Re: [arch-general] Question about automated builder
On Wed, Feb 16, 2011 at 3:12 PM, Magnus Therning wrote: > On Wed, Feb 16, 2011 at 01:10:34PM -0700, Thomas S Hatch wrote: > [..] > > Yes, there sure is, there is a lot of backend code that I am working > > on, but this is coming along! > > > > I am still as much as a few months out from having it production > > ready, and I am a few weeks away from initial functionality testing. > > > > here is the github repo: > > > > https://github.com/thatch45/Quarters > > Great. Let me know if you need someone to feed it packages to build. > I have a few Haskell packages to build each week ;-) > > I'm surprised you're using python, no OCaml? ;-) > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.org jabber: mag...@therning.org > twitter: magthe http://therning.org/magnus > > I invented the term Object-Oriented, and I can tell you I did not have > C++ in mind. > -- Alan Kay > Heh, I like OCaml, but I chose Python for a few reasons, one was so that more people could hack on it, I want it to be very modular, so for instance it will be possible to make a variety of modules to watch different SCM managers, and it wil be possible to write builder daemons in any language. This will make it easy for me to port the system to a faster language if we see the need. Also, python's libs are much more mature, and while I like OCaml a lot, I have written hundreds of thousands of lines of production python, and far less OCaml. I have a few mechanisms to make for package feeding, so for instance, it will watch scm repos for changes and build PKGBUILDs found therein, but I also plan to make an interface that can be used to just feed it source packages. So I will ask for some people to send it source packages to test once I get it going. Also, I will be asking people to try and break it with PKGBUILDS as well, to see if PKGBUILDS can be made that can break out of the build root etc. -Thomas S Hatch
Re: [arch-general] Question about automated builder
On Wed, Feb 16, 2011 at 01:10:34PM -0700, Thomas S Hatch wrote: [..] > Yes, there sure is, there is a lot of backend code that I am working > on, but this is coming along! > > I am still as much as a few months out from having it production > ready, and I am a few weeks away from initial functionality testing. > > here is the github repo: > > https://github.com/thatch45/Quarters Great. Let me know if you need someone to feed it packages to build. I have a few Haskell packages to build each week ;-) I'm surprised you're using python, no OCaml? ;-) /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus I invented the term Object-Oriented, and I can tell you I did not have C++ in mind. -- Alan Kay pgpGAKuliodOQ.pgp Description: PGP signature
Re: [arch-general] Question about automated builder
On Wed, Feb 16, 2011 at 12:54 PM, Nicolás Reynolds wrote: > El 27/01/11 10:36, Thomas S Hatch dijo: > > I have mentioned this subject before on aur-general, but I wanted to open > a > > discussion about it in the broader community. > > > > I have spent a great deal of my career working with Red Hat Linux, as > have > > many in the professional Linux world, and there is one project that is > > amazingly useful for the Red Hat world, and that is Koji. > > > > You can take a look at the web interface for koji here: > > http://koji.fedoraproject.org/koji/ > > > > Basically, koji acts as a continuous build server for the fedora > project's > > rpms. it has a number of benefits, one is that all of the building is > done > > in a controlled environment, so we have a %100 assurance that the > packages > > makedepends are correct, and and the packages are all built inside of > clean > > chroots. > > > > Koji also allows for very simple mass-rebuilds of packages, since all > anyone > > needs to do it increment the release number and koji can pick up the > change > > and build the package. > > > > I have been passively working on a similar project called quarters, but I > > must admit that my motivation is somewhat low not knowing if the project > is > > in demand. So here is my question, do we think that something like this > > would be a benefit to Arch? Is this the type of project that should merit > my > > attention? > > > > Also, if we do think that this would be a good thing to have for Arch, I > > would like feedback on what types of features the system would have and > how > > it would behave. Right now I am following the idea of supporting a > > distributed build system so that we can have any number of build servers > in > > the fray working away to produce Arch packages for us. I am also > attempting > > to build it in such a way that a database is not required and that the > > interface would be amazingly simple (this is Arch after all). This would > > mean that by mearly checking into svn a package would be built, and then > an > > interface would pop up for the right people to sign it off, and once it > has > > been signed off it would move over. > > > > Or at least that is the basic idea I am running on. The code has not come > > very far, but if the Arch community and developers think that this is a > good > > idea then please let me know, my motivation should soar and I will make > Arch > > a super continuous package build system! > > > > -Thomas S Hatch > > -TU > > > > Hi, is there any progress on this? :) > > -- > Salud! > Nicolás Reynolds, > xmpp:fa...@kiwwwi.com.ar > omb:http://identi.ca/fauno > blog:http://selfdandi.com.ar/ > gnu/linux user #455044 > > OTR: C0CB1F0F 01DB5E18 2D634C2A A4626858 E7C7C3A2 > > http://parabolagnulinux.org > http://endefensadelsl.org > Yes, there sure is, there is a lot of backend code that I am working on, but this is coming along! I am still as much as a few months out from having it production ready, and I am a few weeks away from initial functionality testing. here is the github repo: https://github.com/thatch45/Quarters
Re: [arch-general] Question about automated builder
El 27/01/11 10:36, Thomas S Hatch dijo: > I have mentioned this subject before on aur-general, but I wanted to open a > discussion about it in the broader community. > > I have spent a great deal of my career working with Red Hat Linux, as have > many in the professional Linux world, and there is one project that is > amazingly useful for the Red Hat world, and that is Koji. > > You can take a look at the web interface for koji here: > http://koji.fedoraproject.org/koji/ > > Basically, koji acts as a continuous build server for the fedora project's > rpms. it has a number of benefits, one is that all of the building is done > in a controlled environment, so we have a %100 assurance that the packages > makedepends are correct, and and the packages are all built inside of clean > chroots. > > Koji also allows for very simple mass-rebuilds of packages, since all anyone > needs to do it increment the release number and koji can pick up the change > and build the package. > > I have been passively working on a similar project called quarters, but I > must admit that my motivation is somewhat low not knowing if the project is > in demand. So here is my question, do we think that something like this > would be a benefit to Arch? Is this the type of project that should merit my > attention? > > Also, if we do think that this would be a good thing to have for Arch, I > would like feedback on what types of features the system would have and how > it would behave. Right now I am following the idea of supporting a > distributed build system so that we can have any number of build servers in > the fray working away to produce Arch packages for us. I am also attempting > to build it in such a way that a database is not required and that the > interface would be amazingly simple (this is Arch after all). This would > mean that by mearly checking into svn a package would be built, and then an > interface would pop up for the right people to sign it off, and once it has > been signed off it would move over. > > Or at least that is the basic idea I am running on. The code has not come > very far, but if the Arch community and developers think that this is a good > idea then please let me know, my motivation should soar and I will make Arch > a super continuous package build system! > > -Thomas S Hatch > -TU > Hi, is there any progress on this? :) -- Salud! Nicolás Reynolds, xmpp:fa...@kiwwwi.com.ar omb:http://identi.ca/fauno blog:http://selfdandi.com.ar/ gnu/linux user #455044 OTR: C0CB1F0F 01DB5E18 2D634C2A A4626858 E7C7C3A2 http://parabolagnulinux.org http://endefensadelsl.org pgpOXHjEebopk.pgp Description: PGP signature
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 11:28 AM, Thomas S Hatch wrote: > > > On Fri, Jan 28, 2011 at 11:26 AM, Isaac Dupree < > m...@isaac.cedarswampstudios.org> wrote: > >> On 01/28/11 09:32, Jakob Gruber wrote: >> >>> Another aspect of this is security. Right now, any dev / TU could >>> theoretically check in a correct PKGBUILD but upload a binary package >>> with *insert malicious content* in it to the repos with a very low >>> probability of anyone ever noticing. A (mandatory) central build server >>> could guarantee that the package is actually built with the specified >>> publically available PKGBUILD. >>> >>> I'm not a security expert so please call me out if I'm talking nonsense. >>> >> >> You have to trust all servers that are used for building. (and the servers >> need to collectively have enough processing power to build everything!) If >> we take random volunteers then it's not secure. But it can certainly help >> security in certain ways if done right. >> >> ~Isaac >> > > Yes, we cannot take "random" volunteers, but I am confident that we will be > able to find distributed resources that are secure > Ok my fellow Archers, I have a bit of a proposal to chew on, I am not claiming that it is "done" but it should outline my idea. This is still very rough, so go easy on me, honestly I think I have put it together rather quickly and I assume there are holes. If there are places where you want clarity please let me know and I will fill them in. I will have a fresh github project up in the morning. This project is highly compartmentalized, it should be very easy for collaborators to work on individual components. Thank you for your support, I am excited to get this put together! https://wiki.archlinux.org/index.php/Automated_Package_Build_System -Thomas S Hatch
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 11:26 AM, Isaac Dupree < m...@isaac.cedarswampstudios.org> wrote: > On 01/28/11 09:32, Jakob Gruber wrote: > >> Another aspect of this is security. Right now, any dev / TU could >> theoretically check in a correct PKGBUILD but upload a binary package >> with *insert malicious content* in it to the repos with a very low >> probability of anyone ever noticing. A (mandatory) central build server >> could guarantee that the package is actually built with the specified >> publically available PKGBUILD. >> >> I'm not a security expert so please call me out if I'm talking nonsense. >> > > You have to trust all servers that are used for building. (and the servers > need to collectively have enough processing power to build everything!) If > we take random volunteers then it's not secure. But it can certainly help > security in certain ways if done right. > > ~Isaac > Yes, we cannot take "random" volunteers, but I am confident that we will be able to find distributed resources that are secure
Re: [arch-general] Question about automated builder
On 01/28/11 09:32, Jakob Gruber wrote: Another aspect of this is security. Right now, any dev / TU could theoretically check in a correct PKGBUILD but upload a binary package with *insert malicious content* in it to the repos with a very low probability of anyone ever noticing. A (mandatory) central build server could guarantee that the package is actually built with the specified publically available PKGBUILD. I'm not a security expert so please call me out if I'm talking nonsense. You have to trust all servers that are used for building. (and the servers need to collectively have enough processing power to build everything!) If we take random volunteers then it's not secure. But it can certainly help security in certain ways if done right. ~Isaac
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 9:49 AM, C Anthony Risinger wrote: > On Fri, Jan 28, 2011 at 10:26 AM, Thomas S Hatch > wrote: > > On Fri, Jan 28, 2011 at 9:08 AM, C Anthony Risinger >wrote: > > > > But with that said I feel very strongly that my wants as a commercial > user > > of Arch are not on par with the needs of the Arch community in the > manner, > > in fact I would say that my wants from a commercial perspective should be > > thrown out, I don't want my commercial use of Arch to taint the > community, > > it is one of my greatest fears as an Arch TU and contributor. > > no way, don't even think/worry about that. > > commercial needs are what fund a significant amount of "open > development"; your business problem is what really drives this > endeavor, and makes it worthwhile to sink (limited) time resources > toward. everyone else is freely gaining. nothing to be ashamed of in > even the slightest bit imo; most of us (guessing :-) who do open > development are likely employed and fed by doing other, "less open", > development. > > if everything was 100% free software/labor there wouldn't be much > porridge for all the developer bears out there, and this bear would be > in a more lucrative line of work. starving developers don't generate > much software to play with :-) > > C Anthony > Ha! I agree, but with that said, the wants of the one sure as heck don't outweigh the needs of the open source!!
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 10:26 AM, Thomas S Hatch wrote: > On Fri, Jan 28, 2011 at 9:08 AM, C Anthony Risinger wrote: > > But with that said I feel very strongly that my wants as a commercial user > of Arch are not on par with the needs of the Arch community in the manner, > in fact I would say that my wants from a commercial perspective should be > thrown out, I don't want my commercial use of Arch to taint the community, > it is one of my greatest fears as an Arch TU and contributor. no way, don't even think/worry about that. commercial needs are what fund a significant amount of "open development"; your business problem is what really drives this endeavor, and makes it worthwhile to sink (limited) time resources toward. everyone else is freely gaining. nothing to be ashamed of in even the slightest bit imo; most of us (guessing :-) who do open development are likely employed and fed by doing other, "less open", development. if everything was 100% free software/labor there wouldn't be much porridge for all the developer bears out there, and this bear would be in a more lucrative line of work. starving developers don't generate much software to play with :-) C Anthony
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 9:08 AM, C Anthony Risinger wrote: > On Fri, Jan 28, 2011 at 9:51 AM, Thomas S Hatch > wrote: > > > > Jakob, YES! You are spot on here, one of the main motivations behind a > > system like this is security. While I don't think that this is a problem > > with our developers, I do think that it is a potential future problem, > Arch > > is continuing to grow and at an exponential pace. Security of Arch > packages > > is going to be an increasing issue. I don't want to open up the subject > of > > package signing here, but as a side note, a build system could greatly > aid > > aspects of security ranging from quality control to package signing and > > software verification. > > don't know about "exponential" ;-) > > while not perfect by any means, tracking the file list (and possibly > sizes too) might be useful as a loose check for validity; if a package > suddenly has new files or is vastly different from previous builds > there might be an issue (not necessarily malicious either). > > i am kind of working on this same thing actually, but for my own > personal mirror; i have many packages that i need auto built for > several of my netbooks/laptops and VMs. it would be nice if the tool > was flexible enough to be used in this manner (personal/closed loop). > right now i'm about to try some bauerbill + makepkg hackzors... if > anyone has done this already i would love to hear about it in a new > thread, because it will save me time :-) > > C Anthony > To be perfectly honest, a great deal of my motivation stems from the fact that I could really use an automated Arch package build server for my infrastructure at work, I have so many servers running Arch that manually maintaining our private repo is a bit of a pain :) But with that said I feel very strongly that my wants as a commercial user of Arch are not on par with the needs of the Arch community in the manner, in fact I would say that my wants from a commercial perspective should be thrown out, I don't want my commercial use of Arch to taint the community, it is one of my greatest fears as an Arch TU and contributor.
Re: [arch-general] Question about automated builder
On Fri, Jan 28, 2011 at 9:51 AM, Thomas S Hatch wrote: > > Jakob, YES! You are spot on here, one of the main motivations behind a > system like this is security. While I don't think that this is a problem > with our developers, I do think that it is a potential future problem, Arch > is continuing to grow and at an exponential pace. Security of Arch packages > is going to be an increasing issue. I don't want to open up the subject of > package signing here, but as a side note, a build system could greatly aid > aspects of security ranging from quality control to package signing and > software verification. don't know about "exponential" ;-) while not perfect by any means, tracking the file list (and possibly sizes too) might be useful as a loose check for validity; if a package suddenly has new files or is vastly different from previous builds there might be an issue (not necessarily malicious either). i am kind of working on this same thing actually, but for my own personal mirror; i have many packages that i need auto built for several of my netbooks/laptops and VMs. it would be nice if the tool was flexible enough to be used in this manner (personal/closed loop). right now i'm about to try some bauerbill + makepkg hackzors... if anyone has done this already i would love to hear about it in a new thread, because it will save me time :-) C Anthony
Re: [arch-general] Question about automated builder
Thanks Allan, I will look at pacbuild. Jelle, as for AUR interactions, this would not be a primary use by any means, I think that serving the core Arch repos would be top priority. There are a lot of ideas about interacting with the AUR, but I think that resource wise and logistically it should in nowise be considered the primary focus. With that said, I think that making the builder as pluggable as possible will allow us to grow into these roles if we see them as useful in the future. As for the behavior of the build process wrt releases, I think that we are going to have to really iron that out, we have a lot of ideas on how to do it. I think that we need to list out these ideas and, as a community, decide on how we want to move forward. An automated build systems in not just a "convenient way to build packages" it is a quality control gateway for the distribution, a verification checkpoint for package quality and consistency. We need to figure out how to approach this process in light of having such a gateway. Jakob, YES! You are spot on here, one of the main motivations behind a system like this is security. While I don't think that this is a problem with our developers, I do think that it is a potential future problem, Arch is continuing to grow and at an exponential pace. Security of Arch packages is going to be an increasing issue. I don't want to open up the subject of package signing here, but as a side note, a build system could greatly aid aspects of security ranging from quality control to package signing and software verification. All in all the discussion here has brought a lot of questions to light, I am writing up a design proposal on the wiki, but so far it has only touched the software design aspects and not the fact that this system will almost definitely have ramifications on the software release process. Thanks! Keep the comments coming! -Thomas S Hatch
Re: [arch-general] Question about automated builder
Another aspect of this is security. Right now, any dev / TU could theoretically check in a correct PKGBUILD but upload a binary package with *insert malicious content* in it to the repos with a very low probability of anyone ever noticing. A (mandatory) central build server could guarantee that the package is actually built with the specified publically available PKGBUILD. I'm not a security expert so please call me out if I'm talking nonsense.
Re: [arch-general] Question about automated builder
On Thu, 2011-01-27 at 12:34 -0700, Thomas S Hatch wrote: > On Thu, Jan 27, 2011 at 12:28 PM, Thomas Dziedzic wrote: > > > On Thu, Jan 27, 2011 at 1:12 PM, Thomas S Hatch > > wrote: > > > On Thu, Jan 27, 2011 at 12:01 PM, C Anthony Risinger > >wrote: > > > > > >> On Thu, Jan 27, 2011 at 12:37 PM, Thomas S Hatch > > >> wrote: > > >> > On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif > > >> wrote: > > >> > > > >> >> On 28 January 2011 01:36, Thomas S Hatch wrote: > > >> >> > I have been passively working on a similar project called quarters, > > >> but I > > >> >> > must admit that my motivation is somewhat low not knowing if the > > >> project > > >> >> is > > >> >> > in demand. So here is my question, do we think that something like > > >> this > > >> >> > would be a benefit to Arch? Is this the type of project that should > > >> merit > > >> >> my > > >> >> > attention? > > >> >> > > >> >> You have my personal full support. > > >> >> > > >> >> Does this Koji allow people to upload their own .spec/.src packages > > >> >> and offer them a build? I'm thinking something like that for quarters > > >> >> would be good. We can separate the building into 3 categories: > > >> >> > > >> >> == Distribution == > > >> >> This is where devs and TUs connect. If you can work out some kind of > > >> >> integration, it will be totally seamless. Subversion hooks can > > trigger > > >> >> the builds, which then are placed in the respective home folders in > > >> >> gerolde/sigurd. They can be auto uploaded with dbscripts as well but > > I > > >> >> don't know if that's a good idea, mainly because there needs to be > > >> >> inspection (namcap and other habits) before the binary gets served > > >> >> across the mirrors. > > >> >> > > >> >> == Projects == > > >> >> Any third-party packaging initiative can hook up to the system, and > > in > > >> >> turn get their binaries cooked. No-one is responsible for bad > > >> >> packages. > > >> >> > > >> >> == Community == > > >> >> Users submit a PKGBUILD and in turn can download a Pacman package. > > >> >> No-one is responsible for bad packages. > > >> >> > > >> > > > >> > HAHA! I had not thought of that! I love it! The build system can build > > >> user > > >> > packages from uploaded PKGBUILDS, I would need to add some extra > > security > > >> on > > >> > the chroots (or build them in super thin virtual machines), but that > > >> would > > >> > be great, users could verify that their packages were top notch before > > >> > submitting them to the AUR and TUs could check packages much more > > easily. > > >> > > > >> > As for the svn hooks, I was actually looking at polling the scms, this > > >> way > > >> > an scm can be completely detached from the builder and the builder can > > >> just > > >> > arbitrarily build from any old scm. I think that the solution here is > > to > > >> > configure the scms with specific criteria, so that they build into > > >> specific > > >> > repos. > > >> > > > >> > And thanks for your support Ray, it means a lot :) > > >> > > >> did somebody say distributed AUR? > > >> > > >> add a little P2P sharing of the PKG bits into localized repositories > > >> and you got yourself a winner. > > >> > > >> C Anthony > > >> > > > > > > Honestly, a build system could check AUR packages for cleanliness and > > make a > > > binary repo of working clan packages? > > > > > > We have been discussing this in the TU chat, and there is a lot of > > > excitement about it, I am going to post some degign docs on the wiki here > > in > > > a few days (give me some time to put it together :) ) and then we can > > have a > > > free for all on how we want this to work. > > > > > > In the meantime, keep throwing ideas at me so I can work them into the > > > design! > > > > > > Thanks again for the feedback! > > > > > > -Thomas S Hatch > > > > > > > Hey, as I said in the irc, I also have given my full support for this. > > I am willing to work on this also. > > > > I see a lot of potential especially in the aur portion of this system. > > Mainly because it would guarantee that pkgs in the aur are "correct" > > in the sense that they have correct dependencies, makedependencies and > > proper variables (no $startdir) and that then can be built in a clean > > chroot. > > > > Also, just to reiterate what I've said to Thomas Hatch, I have > > implemented an aur clean chroot build system that works recursively: > > https://bbs.archlinux.org/viewtopic.php?id=111366 > > > > Yes, aurtools will prove to be a great asset for the builder. > > FYI, I don't think I mentioned this, so far I have been calling the builder > quarters, since we all know that in the arcade quarters are what really > feeds pacman! > > My project is on google code (I will be moving it to github so that making > it official will be easier) and you are welcome to look it over and look at > me preliminary design doc. > > http://code.google.com/p/quarters/ > > Keep in mind, what is on google code will change dramati
Re: [arch-general] Question about automated builder
Just as an FYI, have a look a the old pacbuild project: http://projects.archlinux.org/pacbuild.git/ I have no idea how far along its development was, but it might give you some ideas.
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 4:53 PM, Allan McRae wrote: > On 28/01/11 03:36, Thomas S Hatch wrote: > >> I have been passively working on a similar project called quarters, but I >> must admit that my motivation is somewhat low not knowing if the project >> is >> in demand. So here is my question, do we think that something like this >> would be a benefit to Arch? Is this the type of project that should merit >> my >> attention? >> > > An automated build system, in particular for the big rebuilds we do, has > been discussed among the developers many times. So there is definitely > interest in such a system. > > > Also, if we do think that this would be a good thing to have for Arch, I >> would like feedback on what types of features the system would have and >> how >> it would behave. Right now I am following the idea of supporting a >> distributed build system so that we can have any number of build servers >> in >> the fray working away to produce Arch packages for us. I am also >> attempting >> to build it in such a way that a database is not required and that the >> interface would be amazingly simple (this is Arch after all). This would >> mean that by mearly checking into svn a package would be built, and then >> an >> interface would pop up for the right people to sign it off, and once it >> has >> been signed off it would move over. >> > > I have never liked the idea of automatic building after a SVN commit. There > a plenty of times that commits are made to SVN in preparation for an update > at a later time. Instead, I would prefer a command-line queue submission > tool (local or remote...). You could have it flexible enough to take a > "makepkg --source" tarball, or allow configuration for updating then > building from a SVN/git/etc repo. > > I agree with this argument, and I have had many thoughts on how to handle it, the reason is because I really like the idea of managing from SVN commits, I think it would make the whole process much easier. What I don't want is an SVN commit to turn into a package push to a production repo, that would be bad. if there is a commit for future use in SVN, and a build happens, thats fine with me, but it could be easily rebuilt when the "future condition" occurs and the process of moving the package from the "build bin" to a production repo should probably always be a manual one. > Thinking about what I do in packaging (and as someone who does not use the > AUR), I see the following situations that need to be covered: > > > 1) single package build: This would probably be the majority use case > > The big idea here is more that a system can do the chroot build and packages checking for the packager, one of the big motivations :) > > 2) multi-package build: Sometimes packages come in pairs/groups where > updating one requires updating the next. e.g. tcl and tk. That requires > building tcl, adding the new package to the chroot and then building tk. > > Note there could be cases where this is more complicated... e.g. Updating > package A,B,C,D. A is required for B, B is required for C and B is > required for D. Note that installing C in the chroot after building would > be unclean for the build of D. Think updates to desktop environments as an > example of this. > > makechrootpkg has the facility to add packages to a local repo after being > built, so that could be used to implement this. > > I have a solution for this, simply that the builders build every package alone in its own chroot, if it has a dep, then the dep needs to be built and added to a repo that the chroot will be able to get to, so more than one package is never built in a single chroot > > 3) big rebuild: e.g. for library soname bump. There are two types of > builds here. A) the initial builds and B) all the packages requiring > rebuild because of those in A. For B, automatic pkgrel bumping would be > good. Also, some automatic determination of the build order for B would be > very helpful. This would require error handling as if a package low in the > dependency chain fails, the rest of that chain should not be built. Again, > adding the packages to a local repo as the rebuild progresses will allow to > build packages cleanly against those that are already rebuilt. Also not > that there could be no packages in list A (e.g. if we just want to rebuild > all of [core]). > > I don't think that this should be a difficult issue, I am a little concerned with auto bumping the pkgrel numbers and committing back to svn, but I am sure that is something I can figure out how to do in a clean and safe way > > 4) flexibility to add makepkg paramaters: I'm not sure how widespread use > this would get... > e.g. the toolchain is built in the following order: > linux-api-headers->glibc->binutils->gcc->binutils->glibc > The first glibc and binutils builds will be built with the --nocheck option > in the future. The gcc and final binutils and glibc builds are built with > -L to save the te
Re: [arch-general] Question about automated builder
On 28/01/11 03:36, Thomas S Hatch wrote: I have been passively working on a similar project called quarters, but I must admit that my motivation is somewhat low not knowing if the project is in demand. So here is my question, do we think that something like this would be a benefit to Arch? Is this the type of project that should merit my attention? An automated build system, in particular for the big rebuilds we do, has been discussed among the developers many times. So there is definitely interest in such a system. Also, if we do think that this would be a good thing to have for Arch, I would like feedback on what types of features the system would have and how it would behave. Right now I am following the idea of supporting a distributed build system so that we can have any number of build servers in the fray working away to produce Arch packages for us. I am also attempting to build it in such a way that a database is not required and that the interface would be amazingly simple (this is Arch after all). This would mean that by mearly checking into svn a package would be built, and then an interface would pop up for the right people to sign it off, and once it has been signed off it would move over. I have never liked the idea of automatic building after a SVN commit. There a plenty of times that commits are made to SVN in preparation for an update at a later time. Instead, I would prefer a command-line queue submission tool (local or remote...). You could have it flexible enough to take a "makepkg --source" tarball, or allow configuration for updating then building from a SVN/git/etc repo. Thinking about what I do in packaging (and as someone who does not use the AUR), I see the following situations that need to be covered: 1) single package build: This would probably be the majority use case 2) multi-package build: Sometimes packages come in pairs/groups where updating one requires updating the next. e.g. tcl and tk. That requires building tcl, adding the new package to the chroot and then building tk. Note there could be cases where this is more complicated... e.g. Updating package A,B,C,D. A is required for B, B is required for C and B is required for D. Note that installing C in the chroot after building would be unclean for the build of D. Think updates to desktop environments as an example of this. makechrootpkg has the facility to add packages to a local repo after being built, so that could be used to implement this. 3) big rebuild: e.g. for library soname bump. There are two types of builds here. A) the initial builds and B) all the packages requiring rebuild because of those in A. For B, automatic pkgrel bumping would be good. Also, some automatic determination of the build order for B would be very helpful. This would require error handling as if a package low in the dependency chain fails, the rest of that chain should not be built. Again, adding the packages to a local repo as the rebuild progresses will allow to build packages cleanly against those that are already rebuilt. Also not that there could be no packages in list A (e.g. if we just want to rebuild all of [core]). 4) flexibility to add makepkg paramaters: I'm not sure how widespread use this would get... e.g. the toolchain is built in the following order: linux-api-headers->glibc->binutils->gcc->binutils->glibc The first glibc and binutils builds will be built with the --nocheck option in the future. The gcc and final binutils and glibc builds are built with -L to save the testsuite build logs. I guess you would make -L the default anyway and have them available at the end of the build for inspection. Anyway, #3 is what I consider most important in an automated build system designed to make the job of packaging in Arch easier. It is also the level where I would start using such a system Building a single package locally is a two "makechrootpkg" commands (one per architecture) and I do not have to transfer the resultant package to test. So I doubt I would use a build system for that (unless build time was massive) as I can not see the automated build being more efficient for me. Allan
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 2:03 PM, Thomas Dziedzic wrote: > On Thu, Jan 27, 2011 at 2:06 PM, Thomas S Hatch > wrote: > > On Thu, Jan 27, 2011 at 12:57 PM, C Anthony Risinger >wrote: > > > >> On Thu, Jan 27, 2011 at 1:48 PM, Thomas S Hatch > >> wrote: > >> > > >> > Awesome, I actually have a few servers I will use, and since it will > be > >> > distributed, we will be able to use a lot of servers as builders all > >> > reporting to a single master. > >> > >> why all the distributed goodness, then a crippling single master? it > >> would not be much more difficult to use 1-to-1 queues-to-server. > >> nodes that fill up return a redirect, or denial. a simple DNS scheme > >> can facilitate auto-discovery... nodes register for an ID under the > >> domain, any simply move down the list. TXT records can provide the > >> lists of root nodes if you really want. with more work, this can be > >> made to work through NAT, a la p2-esque, but that's more ambitious. > >> > >> > As for the web end, I was thinking of having the web frontend just act > as > >> a > >> > notification area about the queue and the builds, so people could > check > >> on > >> > build stats and download experimental pkgs etc. Then the queue would > be > >> > managed via the scms. > >> > >> no way man, go big or go home! web interface is full out AUR > replacement. > >> > >> C Anthony > >> > > > > Hmm, a pure peer setup? My thought on the master has more to do with > central > > job control, and the fact that distributing can be easily done, the > master > > is a lightwieght server compared tot he builders, and the master will > just > > present what needs to be build and pull whats built into repos, this way > the > > master can scale out to a ton of builders and all of the decisions about > who > > builds is done by the builders talking to each other. > > > > Of course we could have separate master dedicated to specific repos > and > > configurations but share the builder pool, so that we have a simple many > to > > many relationship. > > > I prefer the central server idea rather then a purely distributed > system simply because we can't distribute workload well with a purely > distributed one. Imagine serving openoffice and libreoffice to one > server and some python module to another server. At least with the > central server, we have more control over this. > h, I don't think that that will be a problem, my concern is mostly that we would have duplicate packages getting built. A single builder will be able to take hours to build a package and another builder can just spin through say 50 packages in the meantime, also the individual builders will be configurable to do multiple builds at once. But with that said, I am still leaning away from having a set of master peers, I am still chewing on the concept, and how it would be best to do it.
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 2:06 PM, Thomas S Hatch wrote: > On Thu, Jan 27, 2011 at 12:57 PM, C Anthony Risinger wrote: > >> On Thu, Jan 27, 2011 at 1:48 PM, Thomas S Hatch >> wrote: >> > >> > Awesome, I actually have a few servers I will use, and since it will be >> > distributed, we will be able to use a lot of servers as builders all >> > reporting to a single master. >> >> why all the distributed goodness, then a crippling single master? it >> would not be much more difficult to use 1-to-1 queues-to-server. >> nodes that fill up return a redirect, or denial. a simple DNS scheme >> can facilitate auto-discovery... nodes register for an ID under the >> domain, any simply move down the list. TXT records can provide the >> lists of root nodes if you really want. with more work, this can be >> made to work through NAT, a la p2-esque, but that's more ambitious. >> >> > As for the web end, I was thinking of having the web frontend just act as >> a >> > notification area about the queue and the builds, so people could check >> on >> > build stats and download experimental pkgs etc. Then the queue would be >> > managed via the scms. >> >> no way man, go big or go home! web interface is full out AUR replacement. >> >> C Anthony >> > > Hmm, a pure peer setup? My thought on the master has more to do with central > job control, and the fact that distributing can be easily done, the master > is a lightwieght server compared tot he builders, and the master will just > present what needs to be build and pull whats built into repos, this way the > master can scale out to a ton of builders and all of the decisions about who > builds is done by the builders talking to each other. > > Of course we could have separate master dedicated to specific repos and > configurations but share the builder pool, so that we have a simple many to > many relationship. > I prefer the central server idea rather then a purely distributed system simply because we can't distribute workload well with a purely distributed one. Imagine serving openoffice and libreoffice to one server and some python module to another server. At least with the central server, we have more control over this.
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 12:57 PM, C Anthony Risinger wrote: > On Thu, Jan 27, 2011 at 1:48 PM, Thomas S Hatch > wrote: > > > > Awesome, I actually have a few servers I will use, and since it will be > > distributed, we will be able to use a lot of servers as builders all > > reporting to a single master. > > why all the distributed goodness, then a crippling single master? it > would not be much more difficult to use 1-to-1 queues-to-server. > nodes that fill up return a redirect, or denial. a simple DNS scheme > can facilitate auto-discovery... nodes register for an ID under the > domain, any simply move down the list. TXT records can provide the > lists of root nodes if you really want. with more work, this can be > made to work through NAT, a la p2-esque, but that's more ambitious. > > > As for the web end, I was thinking of having the web frontend just act as > a > > notification area about the queue and the builds, so people could check > on > > build stats and download experimental pkgs etc. Then the queue would be > > managed via the scms. > > no way man, go big or go home! web interface is full out AUR replacement. > > C Anthony > Hmm, a pure peer setup? My thought on the master has more to do with central job control, and the fact that distributing can be easily done, the master is a lightwieght server compared tot he builders, and the master will just present what needs to be build and pull whats built into repos, this way the master can scale out to a ton of builders and all of the decisions about who builds is done by the builders talking to each other. Of course we could have separate master dedicated to specific repos and configurations but share the builder pool, so that we have a simple many to many relationship.
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 1:48 PM, Thomas S Hatch wrote: > > Awesome, I actually have a few servers I will use, and since it will be > distributed, we will be able to use a lot of servers as builders all > reporting to a single master. why all the distributed goodness, then a crippling single master? it would not be much more difficult to use 1-to-1 queues-to-server. nodes that fill up return a redirect, or denial. a simple DNS scheme can facilitate auto-discovery... nodes register for an ID under the domain, any simply move down the list. TXT records can provide the lists of root nodes if you really want. with more work, this can be made to work through NAT, a la p2-esque, but that's more ambitious. > As for the web end, I was thinking of having the web frontend just act as a > notification area about the queue and the builds, so people could check on > build stats and download experimental pkgs etc. Then the queue would be > managed via the scms. no way man, go big or go home! web interface is full out AUR replacement. C Anthony
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 12:33 PM, Kaiting Chen wrote: > On Thu, Jan 27, 2011 at 2:12 PM, Thomas S Hatch > wrote: > > > We have been discussing this in the TU chat, and there is a lot of > > excitement about it, I am going to post some degign docs on the wiki here > > in > > a few days (give me some time to put it together :) ) and then we can > have > > a > > free for all on how we want this to work. > > > > Hey dude like I've said before I think this is a great idea. Personally I > would like to see it implemented as a web application where a source can > perform an authenticated multipart HTTP Post to upload the package, at > which > point the service will build the package and return an HTTP 201 status with > the URL of the newly built package in the Location header. Of course this > is > just my opinion and you're free to go about it any way you like. > > I'll be happy to offer up the kiwilight server if ioni doesn't let you have > alderaan. --Kaiting. > > -- > Kiwis and Limes: http://kaitocracy.blogspot.com/ > Awesome, I actually have a few servers I will use, and since it will be distributed, we will be able to use a lot of servers as builders all reporting to a single master. As for the web end, I was thinking of having the web frontend just act as a notification area about the queue and the builds, so people could check on build stats and download experimental pkgs etc. Then the queue would be managed via the scms. Of course, it will be pluggable, so we can make a web "scm" interface as well if we want!
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 2:12 PM, Thomas S Hatch wrote: > We have been discussing this in the TU chat, and there is a lot of > excitement about it, I am going to post some degign docs on the wiki here > in > a few days (give me some time to put it together :) ) and then we can have > a > free for all on how we want this to work. > Hey dude like I've said before I think this is a great idea. Personally I would like to see it implemented as a web application where a source can perform an authenticated multipart HTTP Post to upload the package, at which point the service will build the package and return an HTTP 201 status with the URL of the newly built package in the Location header. Of course this is just my opinion and you're free to go about it any way you like. I'll be happy to offer up the kiwilight server if ioni doesn't let you have alderaan. --Kaiting. -- Kiwis and Limes: http://kaitocracy.blogspot.com/
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 12:28 PM, Thomas Dziedzic wrote: > On Thu, Jan 27, 2011 at 1:12 PM, Thomas S Hatch > wrote: > > On Thu, Jan 27, 2011 at 12:01 PM, C Anthony Risinger >wrote: > > > >> On Thu, Jan 27, 2011 at 12:37 PM, Thomas S Hatch > >> wrote: > >> > On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif > >> wrote: > >> > > >> >> On 28 January 2011 01:36, Thomas S Hatch wrote: > >> >> > I have been passively working on a similar project called quarters, > >> but I > >> >> > must admit that my motivation is somewhat low not knowing if the > >> project > >> >> is > >> >> > in demand. So here is my question, do we think that something like > >> this > >> >> > would be a benefit to Arch? Is this the type of project that should > >> merit > >> >> my > >> >> > attention? > >> >> > >> >> You have my personal full support. > >> >> > >> >> Does this Koji allow people to upload their own .spec/.src packages > >> >> and offer them a build? I'm thinking something like that for quarters > >> >> would be good. We can separate the building into 3 categories: > >> >> > >> >> == Distribution == > >> >> This is where devs and TUs connect. If you can work out some kind of > >> >> integration, it will be totally seamless. Subversion hooks can > trigger > >> >> the builds, which then are placed in the respective home folders in > >> >> gerolde/sigurd. They can be auto uploaded with dbscripts as well but > I > >> >> don't know if that's a good idea, mainly because there needs to be > >> >> inspection (namcap and other habits) before the binary gets served > >> >> across the mirrors. > >> >> > >> >> == Projects == > >> >> Any third-party packaging initiative can hook up to the system, and > in > >> >> turn get their binaries cooked. No-one is responsible for bad > >> >> packages. > >> >> > >> >> == Community == > >> >> Users submit a PKGBUILD and in turn can download a Pacman package. > >> >> No-one is responsible for bad packages. > >> >> > >> > > >> > HAHA! I had not thought of that! I love it! The build system can build > >> user > >> > packages from uploaded PKGBUILDS, I would need to add some extra > security > >> on > >> > the chroots (or build them in super thin virtual machines), but that > >> would > >> > be great, users could verify that their packages were top notch before > >> > submitting them to the AUR and TUs could check packages much more > easily. > >> > > >> > As for the svn hooks, I was actually looking at polling the scms, this > >> way > >> > an scm can be completely detached from the builder and the builder can > >> just > >> > arbitrarily build from any old scm. I think that the solution here is > to > >> > configure the scms with specific criteria, so that they build into > >> specific > >> > repos. > >> > > >> > And thanks for your support Ray, it means a lot :) > >> > >> did somebody say distributed AUR? > >> > >> add a little P2P sharing of the PKG bits into localized repositories > >> and you got yourself a winner. > >> > >> C Anthony > >> > > > > Honestly, a build system could check AUR packages for cleanliness and > make a > > binary repo of working clan packages? > > > > We have been discussing this in the TU chat, and there is a lot of > > excitement about it, I am going to post some degign docs on the wiki here > in > > a few days (give me some time to put it together :) ) and then we can > have a > > free for all on how we want this to work. > > > > In the meantime, keep throwing ideas at me so I can work them into the > > design! > > > > Thanks again for the feedback! > > > > -Thomas S Hatch > > > > Hey, as I said in the irc, I also have given my full support for this. > I am willing to work on this also. > > I see a lot of potential especially in the aur portion of this system. > Mainly because it would guarantee that pkgs in the aur are "correct" > in the sense that they have correct dependencies, makedependencies and > proper variables (no $startdir) and that then can be built in a clean > chroot. > > Also, just to reiterate what I've said to Thomas Hatch, I have > implemented an aur clean chroot build system that works recursively: > https://bbs.archlinux.org/viewtopic.php?id=111366 > Yes, aurtools will prove to be a great asset for the builder. FYI, I don't think I mentioned this, so far I have been calling the builder quarters, since we all know that in the arcade quarters are what really feeds pacman! My project is on google code (I will be moving it to github so that making it official will be easier) and you are welcome to look it over and look at me preliminary design doc. http://code.google.com/p/quarters/ Keep in mind, what is on google code will change dramatically based on your input! With that all said, much of whats there will be replaced!
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 1:12 PM, Thomas S Hatch wrote: > On Thu, Jan 27, 2011 at 12:01 PM, C Anthony Risinger wrote: > >> On Thu, Jan 27, 2011 at 12:37 PM, Thomas S Hatch >> wrote: >> > On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif >> wrote: >> > >> >> On 28 January 2011 01:36, Thomas S Hatch wrote: >> >> > I have been passively working on a similar project called quarters, >> but I >> >> > must admit that my motivation is somewhat low not knowing if the >> project >> >> is >> >> > in demand. So here is my question, do we think that something like >> this >> >> > would be a benefit to Arch? Is this the type of project that should >> merit >> >> my >> >> > attention? >> >> >> >> You have my personal full support. >> >> >> >> Does this Koji allow people to upload their own .spec/.src packages >> >> and offer them a build? I'm thinking something like that for quarters >> >> would be good. We can separate the building into 3 categories: >> >> >> >> == Distribution == >> >> This is where devs and TUs connect. If you can work out some kind of >> >> integration, it will be totally seamless. Subversion hooks can trigger >> >> the builds, which then are placed in the respective home folders in >> >> gerolde/sigurd. They can be auto uploaded with dbscripts as well but I >> >> don't know if that's a good idea, mainly because there needs to be >> >> inspection (namcap and other habits) before the binary gets served >> >> across the mirrors. >> >> >> >> == Projects == >> >> Any third-party packaging initiative can hook up to the system, and in >> >> turn get their binaries cooked. No-one is responsible for bad >> >> packages. >> >> >> >> == Community == >> >> Users submit a PKGBUILD and in turn can download a Pacman package. >> >> No-one is responsible for bad packages. >> >> >> > >> > HAHA! I had not thought of that! I love it! The build system can build >> user >> > packages from uploaded PKGBUILDS, I would need to add some extra security >> on >> > the chroots (or build them in super thin virtual machines), but that >> would >> > be great, users could verify that their packages were top notch before >> > submitting them to the AUR and TUs could check packages much more easily. >> > >> > As for the svn hooks, I was actually looking at polling the scms, this >> way >> > an scm can be completely detached from the builder and the builder can >> just >> > arbitrarily build from any old scm. I think that the solution here is to >> > configure the scms with specific criteria, so that they build into >> specific >> > repos. >> > >> > And thanks for your support Ray, it means a lot :) >> >> did somebody say distributed AUR? >> >> add a little P2P sharing of the PKG bits into localized repositories >> and you got yourself a winner. >> >> C Anthony >> > > Honestly, a build system could check AUR packages for cleanliness and make a > binary repo of working clan packages? > > We have been discussing this in the TU chat, and there is a lot of > excitement about it, I am going to post some degign docs on the wiki here in > a few days (give me some time to put it together :) ) and then we can have a > free for all on how we want this to work. > > In the meantime, keep throwing ideas at me so I can work them into the > design! > > Thanks again for the feedback! > > -Thomas S Hatch > Hey, as I said in the irc, I also have given my full support for this. I am willing to work on this also. I see a lot of potential especially in the aur portion of this system. Mainly because it would guarantee that pkgs in the aur are "correct" in the sense that they have correct dependencies, makedependencies and proper variables (no $startdir) and that then can be built in a clean chroot. Also, just to reiterate what I've said to Thomas Hatch, I have implemented an aur clean chroot build system that works recursively: https://bbs.archlinux.org/viewtopic.php?id=111366
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 12:01 PM, C Anthony Risinger wrote: > On Thu, Jan 27, 2011 at 12:37 PM, Thomas S Hatch > wrote: > > On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif > wrote: > > > >> On 28 January 2011 01:36, Thomas S Hatch wrote: > >> > I have been passively working on a similar project called quarters, > but I > >> > must admit that my motivation is somewhat low not knowing if the > project > >> is > >> > in demand. So here is my question, do we think that something like > this > >> > would be a benefit to Arch? Is this the type of project that should > merit > >> my > >> > attention? > >> > >> You have my personal full support. > >> > >> Does this Koji allow people to upload their own .spec/.src packages > >> and offer them a build? I'm thinking something like that for quarters > >> would be good. We can separate the building into 3 categories: > >> > >> == Distribution == > >> This is where devs and TUs connect. If you can work out some kind of > >> integration, it will be totally seamless. Subversion hooks can trigger > >> the builds, which then are placed in the respective home folders in > >> gerolde/sigurd. They can be auto uploaded with dbscripts as well but I > >> don't know if that's a good idea, mainly because there needs to be > >> inspection (namcap and other habits) before the binary gets served > >> across the mirrors. > >> > >> == Projects == > >> Any third-party packaging initiative can hook up to the system, and in > >> turn get their binaries cooked. No-one is responsible for bad > >> packages. > >> > >> == Community == > >> Users submit a PKGBUILD and in turn can download a Pacman package. > >> No-one is responsible for bad packages. > >> > > > > HAHA! I had not thought of that! I love it! The build system can build > user > > packages from uploaded PKGBUILDS, I would need to add some extra security > on > > the chroots (or build them in super thin virtual machines), but that > would > > be great, users could verify that their packages were top notch before > > submitting them to the AUR and TUs could check packages much more easily. > > > > As for the svn hooks, I was actually looking at polling the scms, this > way > > an scm can be completely detached from the builder and the builder can > just > > arbitrarily build from any old scm. I think that the solution here is to > > configure the scms with specific criteria, so that they build into > specific > > repos. > > > > And thanks for your support Ray, it means a lot :) > > did somebody say distributed AUR? > > add a little P2P sharing of the PKG bits into localized repositories > and you got yourself a winner. > > C Anthony > Honestly, a build system could check AUR packages for cleanliness and make a binary repo of working clan packages? We have been discussing this in the TU chat, and there is a lot of excitement about it, I am going to post some degign docs on the wiki here in a few days (give me some time to put it together :) ) and then we can have a free for all on how we want this to work. In the meantime, keep throwing ideas at me so I can work them into the design! Thanks again for the feedback! -Thomas S Hatch
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 12:37 PM, Thomas S Hatch wrote: > On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif wrote: > >> On 28 January 2011 01:36, Thomas S Hatch wrote: >> > I have been passively working on a similar project called quarters, but I >> > must admit that my motivation is somewhat low not knowing if the project >> is >> > in demand. So here is my question, do we think that something like this >> > would be a benefit to Arch? Is this the type of project that should merit >> my >> > attention? >> >> You have my personal full support. >> >> Does this Koji allow people to upload their own .spec/.src packages >> and offer them a build? I'm thinking something like that for quarters >> would be good. We can separate the building into 3 categories: >> >> == Distribution == >> This is where devs and TUs connect. If you can work out some kind of >> integration, it will be totally seamless. Subversion hooks can trigger >> the builds, which then are placed in the respective home folders in >> gerolde/sigurd. They can be auto uploaded with dbscripts as well but I >> don't know if that's a good idea, mainly because there needs to be >> inspection (namcap and other habits) before the binary gets served >> across the mirrors. >> >> == Projects == >> Any third-party packaging initiative can hook up to the system, and in >> turn get their binaries cooked. No-one is responsible for bad >> packages. >> >> == Community == >> Users submit a PKGBUILD and in turn can download a Pacman package. >> No-one is responsible for bad packages. >> > > HAHA! I had not thought of that! I love it! The build system can build user > packages from uploaded PKGBUILDS, I would need to add some extra security on > the chroots (or build them in super thin virtual machines), but that would > be great, users could verify that their packages were top notch before > submitting them to the AUR and TUs could check packages much more easily. > > As for the svn hooks, I was actually looking at polling the scms, this way > an scm can be completely detached from the builder and the builder can just > arbitrarily build from any old scm. I think that the solution here is to > configure the scms with specific criteria, so that they build into specific > repos. > > And thanks for your support Ray, it means a lot :) did somebody say distributed AUR? add a little P2P sharing of the PKG bits into localized repositories and you got yourself a winner. C Anthony
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 11:24 AM, Ray Rashif wrote: > On 28 January 2011 01:36, Thomas S Hatch wrote: > > I have been passively working on a similar project called quarters, but I > > must admit that my motivation is somewhat low not knowing if the project > is > > in demand. So here is my question, do we think that something like this > > would be a benefit to Arch? Is this the type of project that should merit > my > > attention? > > You have my personal full support. > > Does this Koji allow people to upload their own .spec/.src packages > and offer them a build? I'm thinking something like that for quarters > would be good. We can separate the building into 3 categories: > > == Distribution == > This is where devs and TUs connect. If you can work out some kind of > integration, it will be totally seamless. Subversion hooks can trigger > the builds, which then are placed in the respective home folders in > gerolde/sigurd. They can be auto uploaded with dbscripts as well but I > don't know if that's a good idea, mainly because there needs to be > inspection (namcap and other habits) before the binary gets served > across the mirrors. > > == Projects == > Any third-party packaging initiative can hook up to the system, and in > turn get their binaries cooked. No-one is responsible for bad > packages. > > == Community == > Users submit a PKGBUILD and in turn can download a Pacman package. > No-one is responsible for bad packages. > HAHA! I had not thought of that! I love it! The build system can build user packages from uploaded PKGBUILDS, I would need to add some extra security on the chroots (or build them in super thin virtual machines), but that would be great, users could verify that their packages were top notch before submitting them to the AUR and TUs could check packages much more easily. As for the svn hooks, I was actually looking at polling the scms, this way an scm can be completely detached from the builder and the builder can just arbitrarily build from any old scm. I think that the solution here is to configure the scms with specific criteria, so that they build into specific repos. And thanks for your support Ray, it means a lot :)
Re: [arch-general] Question about automated builder
On 28 January 2011 01:36, Thomas S Hatch wrote: > I have been passively working on a similar project called quarters, but I > must admit that my motivation is somewhat low not knowing if the project is > in demand. So here is my question, do we think that something like this > would be a benefit to Arch? Is this the type of project that should merit my > attention? You have my personal full support. Does this Koji allow people to upload their own .spec/.src packages and offer them a build? I'm thinking something like that for quarters would be good. We can separate the building into 3 categories: == Distribution == This is where devs and TUs connect. If you can work out some kind of integration, it will be totally seamless. Subversion hooks can trigger the builds, which then are placed in the respective home folders in gerolde/sigurd. They can be auto uploaded with dbscripts as well but I don't know if that's a good idea, mainly because there needs to be inspection (namcap and other habits) before the binary gets served across the mirrors. == Projects == Any third-party packaging initiative can hook up to the system, and in turn get their binaries cooked. No-one is responsible for bad packages. == Community == Users submit a PKGBUILD and in turn can download a Pacman package. No-one is responsible for bad packages.
Re: [arch-general] Question about automated builder
On Thu, Jan 27, 2011 at 11:05 AM, Magnus Therning wrote: > On 27/01/11 17:36, Thomas S Hatch wrote: > [...] > > > I have been passively working on a similar project called quarters, but I > > must admit that my motivation is somewhat low not knowing if the project > is > > in demand. So here is my question, do we think that something like this > > would be a benefit to Arch? Is this the type of project that should merit > my > > attention? > > I am not an Arch dev, but I do help maintain a set of packages and I think > this sort of a things would be useful. > > > Also, if we do think that this would be a good thing to have for Arch, I > > would like feedback on what types of features the system would have and > how > > it would behave. Right now I am following the idea of supporting a > > distributed build system so that we can have any number of build servers > in > > the fray working away to produce Arch packages for us. I am also > attempting > > to build it in such a way that a database is not required and that the > > interface would be amazingly simple (this is Arch after all). This would > > mean that by mearly checking into svn a package would be built, and then > an > > interface would pop up for the right people to sign it off, and once it > has > > been signed off it would move over. > > The set of packages I help maintain is kept in a git-repo on github, so the > first feature request would be to *not* tie it closely to a single VCS. > > The second request would be to allow for more than one repo of packages > to be > monitored by the build bot. The use case would be developers staging their > changes in a personal, but still public, branch/clone. > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.org jabber: mag...@therning.org > twitter: magthe http://therning.org/magnus > > I still think that you should apply to be a TU Magnus :) That is good input, and yes, I was planning on making it scm agnostic, I was going to build in support for svn, git, mercurial and the AUR(this would have to be pluggable so that the builder could go into community/extra) The repo thing is in the design too, and I like that, it will need to support repos both private and public, so that we can have things like an automated testing repo with hooks to migrate packages into the main repos. Thanks for the input Magnus!
Re: [arch-general] Question about automated builder
On 27/01/11 17:36, Thomas S Hatch wrote: [...] > I have been passively working on a similar project called quarters, but I > must admit that my motivation is somewhat low not knowing if the project is > in demand. So here is my question, do we think that something like this > would be a benefit to Arch? Is this the type of project that should merit my > attention? I am not an Arch dev, but I do help maintain a set of packages and I think this sort of a things would be useful. > Also, if we do think that this would be a good thing to have for Arch, I > would like feedback on what types of features the system would have and how > it would behave. Right now I am following the idea of supporting a > distributed build system so that we can have any number of build servers in > the fray working away to produce Arch packages for us. I am also attempting > to build it in such a way that a database is not required and that the > interface would be amazingly simple (this is Arch after all). This would > mean that by mearly checking into svn a package would be built, and then an > interface would pop up for the right people to sign it off, and once it has > been signed off it would move over. The set of packages I help maintain is kept in a git-repo on github, so the first feature request would be to *not* tie it closely to a single VCS. The second request would be to allow for more than one repo of packages to be monitored by the build bot. The use case would be developers staging their changes in a personal, but still public, branch/clone. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus signature.asc Description: OpenPGP digital signature
[arch-general] Question about automated builder
I have mentioned this subject before on aur-general, but I wanted to open a discussion about it in the broader community. I have spent a great deal of my career working with Red Hat Linux, as have many in the professional Linux world, and there is one project that is amazingly useful for the Red Hat world, and that is Koji. You can take a look at the web interface for koji here: http://koji.fedoraproject.org/koji/ Basically, koji acts as a continuous build server for the fedora project's rpms. it has a number of benefits, one is that all of the building is done in a controlled environment, so we have a %100 assurance that the packages makedepends are correct, and and the packages are all built inside of clean chroots. Koji also allows for very simple mass-rebuilds of packages, since all anyone needs to do it increment the release number and koji can pick up the change and build the package. I have been passively working on a similar project called quarters, but I must admit that my motivation is somewhat low not knowing if the project is in demand. So here is my question, do we think that something like this would be a benefit to Arch? Is this the type of project that should merit my attention? Also, if we do think that this would be a good thing to have for Arch, I would like feedback on what types of features the system would have and how it would behave. Right now I am following the idea of supporting a distributed build system so that we can have any number of build servers in the fray working away to produce Arch packages for us. I am also attempting to build it in such a way that a database is not required and that the interface would be amazingly simple (this is Arch after all). This would mean that by mearly checking into svn a package would be built, and then an interface would pop up for the right people to sign it off, and once it has been signed off it would move over. Or at least that is the basic idea I am running on. The code has not come very far, but if the Arch community and developers think that this is a good idea then please let me know, my motivation should soar and I will make Arch a super continuous package build system! -Thomas S Hatch -TU
