Re: [gentoo-dev] RFC - Gentoo on the Lab
On Tue, 2005-08-23 at 12:27 -0500, Brian Harring wrote: > On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote: > > On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote: > > > That and help would always be welcome :P > > > > Then where do I find the code (I'm an official dev yet, so I only have > > access to what's in the mirrors and the patchs on mailing lists) > If you're after a snapshot, give a yell; right now still waiting for > svn crap to straighten out (mainly for the rewrite to be moved fully > to svn), but once done I can point you at wherever I dump it in my > devspace till anonsvn is up. Yeah, I'd really like having a snapshot, even if I'd prefer having cvs/svn access. You can send me it by mail or make it available somewhere. > No clue on eta of anonsvn, that's infra's thing unfortunately. > ~harring -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
Lot of text left inline, pardon, just scroll and deal with it :P On Tue, Aug 23, 2005 at 12:28:08PM -0400, Kristian Benoit wrote: > Here is my recent communication with Pieter: > > On Sat, 2005-08-13 at 21:59 +0200, Pieter Van den Abeele wrote: > > On 13 Aug 2005, at 19:16, Kristian Benoit wrote: > > > > > I've heard that you might be the last to know something about > > > portage-ng. What's the actual status, > > > > Here's what I've done so far: > > > > I've build upon Andreas' Zeller idea of using Smolka's feature logic > > for software configuration management. Zeller's approach was ok for > > determining consistency/inconsistency of software configurations. In > > his phd thesis he described the algorithm involved and discussed time > > complexity (which goes to NP if you allow for quantification). My > > impression after implementing his idea was that for automated > > construction there were a few things that were lacking, and some > > things were incorrect. I revisited Zellers feature logic, and ended > > up with a clean implementation of a declarative language which has > > some very desirable properties for automated software construction. > > I'd say my approach is closer to the spirit of Smolka's feature logic > > than Zellers approach. My non-destructive feature unification has a > > worst case time complexity of O(n x m) where n is the length of the > > feature term describing your system, and m is the length of the > > feature term describing the component to be added. In practice > > performance is very good. An emerge --vp --emptytree kde with the > > regular portage takes about 55 seconds on my Open Desktop Workstation > > and produces a list of 200 packages. A similar action using my > > implementation: > > > > = > > Total time: 14.55 seconds > > = > > Predicate Box Entries =Calls+Redos Time > > = > > vunify/4341,900 = 341,900+072.6% > > $garbage_collect/1 326 = 326+013.6% > > lists:append/3 684,320 = 684,320+0 4.0% > > genterm/2 520 = 520+0 3.9% > > hunify/4520 = 520+0 3.3% > > is/2342,420 = 342,420+0 1.8% > > >/2 342,160 = 342,160+0 0.8% > > buildsystem/2 1 =1+0 0.0% > > val/3 5,200 =5,200+0 0.0% > > unify/3 260 = 260+0 0.0% > > > > % 9,531,861 inferences, 13.96 CPU in 14.55 seconds (96% CPU, 682798 > > Lips) Stating it as nicely as possible, without knowing what that's doing, stats can be construed several ways; faster backend access (both vdb and ebuild/cache), dodging/caching virtuals, etc; language implementation is a point of curiousity also. Faster implementation doesn't surprise me- stable portage is fricking *absolutely* retarded about caching, parsing of deps and cpvs, loading of profile, etc. Either way, interest remains in seeing it :) > > The search space is kept as small as possible because I've introduced > > lazy feature evaluation and both multi valued and open features. > > Those concepts are missing in Zellers approach. Comparing current > > portage and my implementation performance-wise is difficult. In > > general mine is faster, but current portage doesn't use sql either. > > What is for sure is that mine allows you to express various > > constraints. You can prevent a dependency from being built with a > > specific use flag. My implementation automatically solves 'blockers'. > > Circular dependencies are also solved correctly. For instance, if you > > want to install unixodbc with the qt use flag enabled and you want to > > install qt wit the unixodbc use flag enabled, my portage knows that > > it needs to: > > > > emerge unixodbc without qt > > emerge qt with unixodbc > > remerge unixodbc with qt support > > > > So it makes revdep-rebuild unnecessary basically. revdep-rebuild is still necessary, regardless of use deps or full state graphs- anyone doubting that should take a look at the breakage that has occured whenever mysql/ssl have decided to change their soname while maintaining compatibility. Need a bincompat metadata attribute to kill off revdep-rebuild. > > Once I was done implementing this new declarative language in which > > for instance ebuild concepts can be easily expressed (but also rpm, > > deb, fink, darwinports). This sounds a lot like my restriction subsystem in the rewrite (code's available to anyone after it). With the restricti
Re: [gentoo-dev] RFC - Gentoo on the Lab
On 08/23/05 Ricardo Loureiro wrote: > On Mon, 22 Aug 2005 21:41:35 +0100 > Stephen Bennett <[EMAIL PROTECTED]> wrote: > > > portage-ng is dead. There is a rewrite going on, but it'll take a > > while > > to get anywhere near usable. > > I searched a bit to find information about portage-ng but the only > doc I found was Daniel's pdf with just a drawing of the arch itself, > nothing very useful to me. Also the latest news go back almost 2 > years. Is there a place with more information on what portage-ng > should be? Nope. It was basically one of Daniels naive visions mixed with Pieters science blabla (nothing against science, but I wouldn't call a package repository a knowledge base). If there is any actual information in portage-ng, you have to ask Pieter, but officially it's dead (was never alive really), but don't hope to get anything more than "it's using advanced science, it works and I'll release the code someday". Marius -- Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote: > On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote: > > That and help would always be welcome :P > > Then where do I find the code (I'm an official dev yet, so I only have > access to what's in the mirrors and the patchs on mailing lists) If you're after a snapshot, give a yell; right now still waiting for svn crap to straighten out (mainly for the rewrite to be moved fully to svn), but once done I can point you at wherever I dump it in my devspace till anonsvn is up. No clue on eta of anonsvn, that's infra's thing unfortunately. ~harring pgpFq2SET6KV1.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Tue, Aug 23, 2005 at 12:25:03PM -0400, Kristian Benoit wrote: > On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote: > > That and help would always be welcome :P > > Then where do I find the code (I'm an official dev yet, so I only have > access to what's in the mirrors and the patchs on mailing lists) Timing mildy sucks; just switched over to svn, making our anoncvs access pointless for the rewrite. So... after anon svn to replace loss of anoncvs provided by carpaski. Meantime, anyone who wants snapshots can scream at me and I'll post them- anon svn if it is agreed to be implemented infra wise, once it's up the info will be posted in gentoo-portage-dev ml, and tagged into /topic in irc (the norms). ~harring pgp1AfM7QPS5I.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 22 Aug 2005 15:39:10 -0500 Brian Harring <[EMAIL PROTECTED]> wrote: > The 2.1 code that was pushed out for inspection addresses the cache > issue mostly, and modularization as much as possible. Everything > else > falls to the rewrite which is underway- I'd suggest contacting > portage > devs, since what you're after is pretty much what's been designed > to > allow for, without requiring hacks to portage- just would be > plugins. Writing a plugin for portage would be perfect, but I can't wait for 2.1 to be released due to deadlines. Looks like the best shot at this point is a new package using as much as possible the actual python libs created for portage. When I start the project itself (September 1st) I'm gonna start to write and organize all the ideas, but you all have been a great help so far. Ricardo Loureiro -- http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0 pgpC5noQilEE6.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 22 Aug 2005 21:41:35 +0100 Stephen Bennett <[EMAIL PROTECTED]> wrote: > portage-ng is dead. There is a rewrite going on, but it'll take a > while > to get anywhere near usable. I searched a bit to find information about portage-ng but the only doc I found was Daniel's pdf with just a drawing of the arch itself, nothing very useful to me. Also the latest news go back almost 2 years. Is there a place with more information on what portage-ng should be? Ricardo Loureiro -- http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0 pgpV7ARwSYQ0z.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 2005-08-22 at 21:41 +0100, Stephen Bennett wrote: > On Mon, 22 Aug 2005 13:49:14 -0400 > Kristian Benoit <[EMAIL PROTECTED]> wrote: > > > I do agree with that, portage probably need a rewrite/better > > modularization anyway. There is/was a project called portage-ng () you > > might want to have a look at. I did a little in that direction > > recently, and it seems that there is not too many people working on > > it since drobbins left, but you can contact Pieter > > ([EMAIL PROTECTED]). I might get on that too at some point in the > > future too. > > portage-ng is dead. There is a rewrite going on, but it'll take a while > to get anywhere near usable. Here is my recent communication with Pieter: On Sat, 2005-08-13 at 21:59 +0200, Pieter Van den Abeele wrote: > On 13 Aug 2005, at 19:16, Kristian Benoit wrote: > > > I've heard that you might be the last to know something about > > portage-ng. What's the actual status, > > Here's what I've done so far: > > I've build upon Andreas' Zeller idea of using Smolka's feature logic > for software configuration management. Zeller's approach was ok for > determining consistency/inconsistency of software configurations. In > his phd thesis he described the algorithm involved and discussed time > complexity (which goes to NP if you allow for quantification). My > impression after implementing his idea was that for automated > construction there were a few things that were lacking, and some > things were incorrect. I revisited Zellers feature logic, and ended > up with a clean implementation of a declarative language which has > some very desirable properties for automated software construction. > I'd say my approach is closer to the spirit of Smolka's feature logic > than Zellers approach. My non-destructive feature unification has a > worst case time complexity of O(n x m) where n is the length of the > feature term describing your system, and m is the length of the > feature term describing the component to be added. In practice > performance is very good. An emerge --vp --emptytree kde with the > regular portage takes about 55 seconds on my Open Desktop Workstation > and produces a list of 200 packages. A similar action using my > implementation: > > = > Total time: 14.55 seconds > = > Predicate Box Entries =Calls+Redos Time > = > vunify/4341,900 = 341,900+072.6% > $garbage_collect/1 326 = 326+013.6% > lists:append/3 684,320 = 684,320+0 4.0% > genterm/2 520 = 520+0 3.9% > hunify/4520 = 520+0 3.3% > is/2342,420 = 342,420+0 1.8% > >/2 342,160 = 342,160+0 0.8% > buildsystem/2 1 =1+0 0.0% > val/3 5,200 =5,200+0 0.0% > unify/3 260 = 260+0 0.0% > > % 9,531,861 inferences, 13.96 CPU in 14.55 seconds (96% CPU, 682798 > Lips) > > The search space is kept as small as possible because I've introduced > lazy feature evaluation and both multi valued and open features. > Those concepts are missing in Zellers approach. Comparing current > portage and my implementation performance-wise is difficult. In > general mine is faster, but current portage doesn't use sql either. > What is for sure is that mine allows you to express various > constraints. You can prevent a dependency from being built with a > specific use flag. My implementation automatically solves 'blockers'. > Circular dependencies are also solved correctly. For instance, if you > want to install unixodbc with the qt use flag enabled and you want to > install qt wit the unixodbc use flag enabled, my portage knows that > it needs to: > > emerge unixodbc without qt > emerge qt with unixodbc > remerge unixodbc with qt support > > So it makes revdep-rebuild unnecessary basically. > > > Once I was done implementing this new declarative language in which > for instance ebuild concepts can be easily expressed (but also rpm, > deb, fink, darwinports). I implemented a knowledge base in which > those feature terms can be stored/looked up efficiently. I used an > odbc bridge design pattern and have tested the thing with postgresql/ > mysql and sqlite. I've also implemented a DCG parser which parses > ebuilds (no more having to start a bash process for each ebuild like > current portage does) and converts it to feature logic representation > very efficiently. One of the things I'm doing now is finishing up the
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 2005-08-22 at 15:39 -0500, Brian Harring wrote: > That and help would always be welcome :P Then where do I find the code (I'm an official dev yet, so I only have access to what's in the mirrors and the patchs on mailing lists) -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, Aug 22, 2005 at 01:49:14PM -0400, Kristian Benoit wrote: > On Mon, 2005-08-22 at 16:38 +0200, Marius Mauch wrote: > > > Anyway, I hope you realize that your project doesn't only involve > > hacking on portage, but rewriting almost all of it for the client part. > > Actually I'd rather suggest you start from scratch > > I do agree with that, portage probably need a rewrite/better > modularization anyway. There is/was a project called portage-ng () you > might want to have a look at. I did a little in that direction recently, > and it seems that there is not too many people working on it since > drobbins left, but you can contact Pieter ([EMAIL PROTECTED]). I might > get on that too at some point in the future too. Portage-ng never resulted in anything tangible (read: code), further the doc wasn't really useful for anything then jotting down what's desired. Unless something's changed, that doc should've been yanked down. She's dead, jim. Regarding modularization of portage, it requires that, but fundamentally it requires a rewrite of the core; there is no internal package abstraction, repository abstraction, hell, even a clean config abstraction (let alone cache abstraction). The 2.1 code that was pushed out for inspection addresses the cache issue mostly, and modularization as much as possible. Everything else falls to the rewrite which is underway- I'd suggest contacting portage devs, since what you're after is pretty much what's been designed to allow for, without requiring hacks to portage- just would be plugins. That and help would always be welcome :P ~harring pgpTyRVwDeVAP.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 22 Aug 2005 13:49:14 -0400 Kristian Benoit <[EMAIL PROTECTED]> wrote: > I do agree with that, portage probably need a rewrite/better > modularization anyway. There is/was a project called portage-ng () you > might want to have a look at. I did a little in that direction > recently, and it seems that there is not too many people working on > it since drobbins left, but you can contact Pieter > ([EMAIL PROTECTED]). I might get on that too at some point in the > future too. portage-ng is dead. There is a rewrite going on, but it'll take a while to get anywhere near usable. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 2005-08-22 at 16:38 +0200, Marius Mauch wrote: > Anyway, I hope you realize that your project doesn't only involve > hacking on portage, but rewriting almost all of it for the client part. > Actually I'd rather suggest you start from scratch I do agree with that, portage probably need a rewrite/better modularization anyway. There is/was a project called portage-ng () you might want to have a look at. I did a little in that direction recently, and it seems that there is not too many people working on it since drobbins left, but you can contact Pieter ([EMAIL PROTECTED]). I might get on that too at some point in the future too. There is also Patrick Mclean ([EMAIL PROTECTED]) that has a nice lab setup using Gentoo at McGill. It's a diskless system so might not be to much what you are looking for, but as you probably want to dig before getting to work, you should really see his setup. He did not released anything yet, but we talked about starting a wiki for gentoo labs sometime and have his setup released there. Kristian -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
Ricardo Loureiro wrote: Usable in the way that the client machines should be able to use portage, except it's the hacked (or new package) version that should do everything from the SQL server. For example, a emerge package would behave in 2 possible ways;1- calculate it's dependencies from the portage tree on the SQL server and request the binary packages, 2- Request the package and the server would calculate dependencies and get the binary done. I'm more keen on the second since it takes away processor time from the clients, but that involves sending sensitive information such as world files and make.conf over the network. Sounds like in your setup you would like to keep a profile of the client on the server, so you don't have to send over that information, because it is already in the DBMS. That also allows you to update 'push-driven', because the server can easily tell which clients have packages that are out of date. -- Fabian Groffen -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Mon, 22 Aug 2005 16:38:11 +0200 Marius Mauch <[EMAIL PROTECTED]> wrote: > Define "usable". As only portage uses the tree it would be the only > thing that might break. Usable in the way that the client machines should be able to use portage, except it's the hacked (or new package) version that should do everything from the SQL server. For example, a emerge package would behave in 2 possible ways;1- calculate it's dependencies from the portage tree on the SQL server and request the binary packages, 2- Request the package and the server would calculate dependencies and get the binary done. I'm more keen on the second since it takes away processor time from the clients, but that involves sending sensitive information such as world files and make.conf over the network. > As far as I know, yes. But it wasn't what you wanted anyway (only > implemented a SQL cache for faster searching, interesting that > almost > every "rewrite" attempt implements searching first) I asked because it would be nice to talk to their devs, so I could know in advance what were their problems and what they would have done different. Anyway the project will be different, as you said, but another goal was to produce a single machine mode that would use a relational database engine as portage tree. > > I'd guess baselayout + it's deps + libc are the absolute minimum > (excluding baselayout-lite and other embedded solutions). Thanks, that was exactly what I was looking for. > > > 4- Any ideas on how the conf files should be handled? > > Depends on your client nodes, if they are (almost) identical I'd > just > sync them from a master node. If not it gets complicated. That's the problem, very different machines (and maybe some time later even arch's). The best way was to produce yet another etc-update, but 5 months for a single person is too little time for that. In general most of the times if the config files are not changed it's safe to overwrite, else don't, but sometimes pakcage versions have config files re-written, and that's a problem. Just wanted to know what you ppl do in these situations and maybe found something I was not aware of. > > Anyway, I hope you realize that your project doesn't only involve > hacking on portage, but rewriting almost all of it for the client > part. > Actually I'd rather suggest you start from scratch (so you also > make it > work completely without a tree), or wait for Brians rewrite in HEAD > (not > a good idea though if you have a deadline). Server should be less > of an > issue, mostly config tweaks there. My initial thought was a from scratch portage in python that could use many of the code already done, that would be better since portage itself doesn't need a client-server mode and I could learn a lot more this way. Waiting is not an option, no pressure on other ppl and limited time for the project, but I hope to have the time to change it after the deadline as a personal hobbie. > But as Donnie said, gentoo-portage-dev is the better list for this > discussion. Did already, thanks for your help already. Ricardo Loureiro -- http://pgp.dei.uc.pt:11371/pks/lookup?op=get&search=0x6B7C0EC0 pgpdiIpQMZHcF.pgp Description: PGP signature
Re: [gentoo-dev] RFC - Gentoo on the Lab
On 08/22/05 Ricardo Loureiro wrote: > 1- Can I RSYNC_EXCLUDE everything except profiles and have an usable > system? Define "usable". As only portage uses the tree it would be the only thing that might break. > 2- There was a portagesql effort, is it dead? As far as I know, yes. But it wasn't what you wanted anyway (only implemented a SQL cache for faster searching, interesting that almost every "rewrite" attempt implements searching first) > 3- If I tweak a custom profile not to have dev packages such as gcc, > what kind of problems should I expect? Which packages from the base > profile MUST exist? The idea is that the client machines should not > be capable of compiling software. I'd guess baselayout + it's deps + libc are the absolute minimum (excluding baselayout-lite and other embedded solutions). > 4- Any ideas on how the conf files should be handled? Depends on your client nodes, if they are (almost) identical I'd just sync them from a master node. If not it gets complicated. Anyway, I hope you realize that your project doesn't only involve hacking on portage, but rewriting almost all of it for the client part. Actually I'd rather suggest you start from scratch (so you also make it work completely without a tree), or wait for Brians rewrite in HEAD (not a good idea though if you have a deadline). Server should be less of an issue, mostly config tweaks there. But as Donnie said, gentoo-portage-dev is the better list for this discussion. Marius -- Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
On Sun, 2005-08-21 at 20:34 -0700, Donnie Berkholz wrote: > | The final result of the project will be released to the community > | (GPL or BSD, still need to think), so I'd love to ear from users > | dealing with this kind of scenario, question, comments, whatever you > | think I should focus on. Oh, a name for the project would be welcome > | =) > > I strongly encourage you to release it under the same license as Portage > is (GPL-2), so parts (or all) of it could be incorporated into Gentoo. > If portage is GPL-2 it he is building a derivative work does he have a choice really ? -- Cheers, Ivan Yosifov. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ricardo Loureiro wrote: | As part of the final project for my graduation in Informatics | Engineering (kinda Computer Science but that's the official name), | I'm gonna develop a "distributed" portage so we can have a test | lab at our uni with Gentoo (Starting September 1st). You might want to get on the gentoo-portage-dev list to discuss this in more detail to the people most relevant and interested in it. | The final result of the project will be released to the community | (GPL or BSD, still need to think), so I'd love to ear from users | dealing with this kind of scenario, question, comments, whatever you | think I should focus on. Oh, a name for the project would be welcome | =) I strongly encourage you to release it under the same license as Portage is (GPL-2), so parts (or all) of it could be incorporated into Gentoo. Thanks, Donnie -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQFDCUfiXVaO67S1rtsRAmZaAJ0ct39eYsPB3mXEkCRElhBBFN7bCgCgtU2s K+LVTrh5LIaaQEUQQRmPZeQ= =iqZX -END PGP SIGNATURE- -- [EMAIL PROTECTED] mailing list
Re: [gentoo-dev] RFC - Gentoo on the Lab
Ricardo Loureiro wrote: Some questions: 1- Can I RSYNC_EXCLUDE everything except profiles and have an usable system? Depending on exactly how you're doing this, you'll probably want to keep eclasses/ as well. 2- There was a portagesql effort, is it dead? Yes. It has been for a while, afaik. 3- If I tweak a custom profile not to have dev packages such as gcc, what kind of problems should I expect? Which packages from the base profile MUST exist? The idea is that the client machines should not be capable of compiling software. You'll probably run into an ebuild somewhere that directly depends on gcc and/or related dev packages (very likely one of the leftover 'system' packages). 4- Any ideas on how the conf files should be handled? Nope :) -- Andrew Gaffneyhttp://dev.gentoo.org/~agaffney/ Gentoo Linux Developer Installer Project -- [EMAIL PROTECTED] mailing list
