Re: ports system woes
[EMAIL PROTECTED] píše v po 31. 03. 2008 v 01:23 -0700: > > Quick solution would be to gather all depnames for the deleted package, > > and then do a single pass over /var/db/pkg entries looking for origins. > > > > Ultimate solution would be to implement a database which would > > concentrate origins for all packages with linear lookup time. > > I'm personally against of decresing complexity by means of incresing > complexity of pkg_install suit (i.e., database instead of fs). > I might have a distorted understanding, but UFS2 is an OK database in > itself? Can't the the tools be made O(n) without resorting to moving > everything to a DB? You can't query, for example, which package installed some file, without reading the complete +CONTENTS for all installed packages. With, say, SQL database, you could do that under 10 blockreads. > FS-based approach performs well enough when /var/db/pkg/ is cached, so > IMO the issue is just that whole pkg/ tree doesn't fit in cache on > machines with 256M RAM anymore (because of sheer monstrosity of the tree > caused by them 'modular' approaches to large projects). Not much we can do about that, we usually follow the rule one vendor distfile = one port and it has served us well. I would be strongly against merging X.org ports back into a monolitic port again. > I am probably wrong (didn't check in detail), but doesn't bsd.openssl.mk > get invoked every time (every port install) if USE_OPENSSL==yes, no matter > if the openssl port is installed or not? Anyway, it'd be nice if dinoex > could take a look at bsd.openssl.mk (does he need to be CC'd?) bsd.openssl.mk does get included every time USE_OPENSSL is specified in a port, but that's not a problem. The problem is execution of != assignments you pointed out, and these happen only when openssl port is actually installed. -- Pav Lucistnik <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The Novice rogue. A rather shifty individual signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: ports system woes
[EMAIL PROTECTED] píše v po 31. 03. 2008 v 01:00 -0700: > for all this GNOME/GTK stuff? Insane, I say. Look at the depndencies field > for `make search` results -- atrocious. How often do you care about a port > depending on xproto, Xlib, x..., instead of just Xorg? Actually there's an ongoing effort to change ports to depend only on the Xorg pieces that are actually needed, instead on the whole bowl. > Since nothing can be done about it, how about placing all the packages > that constitute base of Xorg into a separate directory in /var/db/pkg/ ? I don't see how that would simplify anything. /var/db/pkg is not ment to be user friendly. -- Pav Lucistnik <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> Pilot to Co-Pilot: Boo. signature.asc Description: Toto je digitálně podepsaná část zprávy
Re: ports system woes
> The pkg database in /var/db/pkg stores two-way dependency chains. > Each port lists all it's dependencies in +CONTENTS file, and all ports > that depend on it in +REQUIRED_BY. When you delete package, all > dependencies of deleted package are iterated and the name of deleted > package is removed from dependency's +REQUIRED_BY file. That's what > undepend() do. aha, thanks! hmm, this should have been pretty obvious to me... > Quick solution would be to gather all depnames for the deleted package, > and then do a single pass over /var/db/pkg entries looking for origins. > > Ultimate solution would be to implement a database which would > concentrate origins for all packages with linear lookup time. I'm personally against of decresing complexity by means of incresing complexity of pkg_install suit (i.e., database instead of fs). I might have a distorted understanding, but UFS2 is an OK database in itself? Can't the the tools be made O(n) without resorting to moving everything to a DB? FS-based approach performs well enough when /var/db/pkg/ is cached, so IMO the issue is just that whole pkg/ tree doesn't fit in cache on machines with 256M RAM anymore (because of sheer monstrosity of the tree caused by them 'modular' approaches to large projects). > The OpenSSL thing I assume is only relevant for people who happen to > have OpenSSL installed from ports. For that, it could be solved by > spamming the required value into /etc/make.conf, similar what perl ports > do. But that really is up to the openssl port maintainer > ([EMAIL PROTECTED]). right, but note how I suggested 'intelligently modifying' rather than "spamming" :) I am probably wrong (didn't check in detail), but doesn't bsd.openssl.mk get invoked every time (every port install) if USE_OPENSSL==yes, no matter if the openssl port is installed or not? Anyway, it'd be nice if dinoex could take a look at bsd.openssl.mk (does he need to be CC'd?) > Pav Lucistnik <[EMAIL PROTECTED]> > <[EMAIL PROTECTED]> [SorAlx] ridin' VS1400 ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
> On Wed, 26 Mar 2008 14:18:00 +0100, Michel Talon wrote > > In fact last year i wrote a python script which reads all the > > /var/db/pkg/+CONTENTS files, and fixes all the +REQUIRED_BY files, > > assuming they are corrupted. Moreover it follows the MOVED file. > > So you basically reimplemented pkgdb -F in python? > > > As far as i remember this program > > runs in a few *seconds* certainly not minutes like it is said here > > Mind that the original poster is using a very low-spec notebook with next > to none RAM. yes, and every developer should test their creations on something like this :) something even slower in fact -- think PDA-like device or UMPC. More seriously, though -- it's a midrange machine, it's fast enough for most development tasks (think CAD, Eagle, etc), and when you think about it, 256Mbytes of RAM is plenty. I believe the issue here is not that this machine's RAM is too small, but /var/db/pkg is far too freakin' huge. Over 300 entries for an X11 system? Don't make me laugh. A gazillion more for all this GNOME/GTK stuff? Insane, I say. Look at the depndencies field for `make search` results -- atrocious. How often do you care about a port depending on xproto, Xlib, x..., instead of just Xorg? Since nothing can be done about it, how about placing all the packages that constitute base of Xorg into a separate directory in /var/db/pkg/ ? GNOME stuff could go into another dir. Ditto for other large projects, like KDE. The rest could stay as it is (thereby preserving the incredible convenience and ease with which packages can be managed -- i.e., the UNiX way of text- and filesystem-based shell tools). This would also clean up /var/db/pkg quite a bit -- that is to say, one could more easily have a look over which [useful|non-useful] packages are installed, without diggin' in a directory with hundreds of subdirs. > > solution is to use sqlite and not some half-assed solution like a > > Berkeley database, > > Solution is to use tools that are available in our base system. SQLite is > not. I remember there being a thread about this some time ago, but I don't remember -- why can't it be made part of our base? In two words... > Pav Lucistnik <[EMAIL PROTECTED]> > <[EMAIL PROTECTED]> [SorAlx] ridin' VS1400 ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
On Thu, Mar 27, 2008 at 10:33:44AM +0100, Michel Talon wrote: >Garrett Cooper wrote: >> We came to the conclusion that BDB should be used, as no other DB >> backend / API exists in the base system (currently), and porting >> SQLLite (while nice) appeared to be non-trivial to port > >Are you kidding? The patch files are totally trivial modifications, >to include stdlib.h. The bigger one is in Makefile.in to take into >account these ones. OTOH, SQLite is currently a rapidly moving target. And a couple of versions have including warnings about backward/forward compatability. Both these points count against integrating SQLite into the base system. Note that one of the reasons for removing perl from the base system was the difficulty of tracking vendor changes. >However it should not be bad to evaluate a solution based on BerkeleyDB, >another one on sqlite, and chose based on merit, not on aversions. This is reasonable. Of course, this is probably 50% more effort than just doing one - and no-one has come forward with a single solution yet. >particular an obvious fact is that there are constant troubles when the >DB version number changes or the ruby adapter changes. If you use the base BDB1 then it never changes, though portupgrade can become very upset when it upgrades its own dependencies. > One may expect >that no such problems will occur with a very stable and standardized >language like it is offered by sqlite. SQL is standardised but its implementation in SQLite is not yet stable. > It would be >useful to compare what the BdB in the base system has to offer >compared to sqlite BDB1 only offers get/put/delete. There is no locking or atomicity (these are only available in the ports version). -- Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. pgptHxSELRShl.pgp Description: PGP signature
Re: ports system woes
Michel Talon writes: > > and got a lot of unhappy responses from folks. > > This is true. A lot of people expressed aversion against SQL, by > itself. As I remember it ... there was some aversion to SQLite. There was stronger aversion to: 1) using anything not in the base system unless it gives substantial payoff 1a) adding something to the base system just for one project, no matter how important that project may be 2) a system which cannot be fixed with the use of a basic text editor and less than guru level knowledge Now, it may be possible the current system also fails (2). But any replacement which make that worse (as I remember was claimed for SQLite) is fighting an uphill battle. Robert Huff ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
Garrett Cooper wrote: > We're rehashing the discussion made last year around June - July. Indeed. > > We came to the conclusion that BDB should be used, as no other DB > backend / API exists in the base system (currently), and porting > SQLLite (while nice) appeared to be non-trivial to port Are you kidding? The patch files are totally trivial modifications, to include stdlib.h. The bigger one is in Makefile.in to take into account these ones. > and got a lot of unhappy responses from folks. This is true. A lot of people expressed aversion against SQL, by itself. However it should not be bad to evaluate a solution based on BerkeleyDB, another one on sqlite, and chose based on merit, not on aversions. What is the simplest to use by the programmer writing pkg_* tools, what offers the best performance and data organization, etc. At the moment portupgrade uses a BerkeleyDB, are you convinced by the result? In particular an obvious fact is that there are constant troubles when the DB version number changes or the ruby adapter changes. One may expect that no such problems will occur with a very stable and standardized language like it is offered by sqlite. If this argument is correct, it is quite strong, in my opinion, because i don't expect much performance difference otherwise. There is also the question of atomicity and locking which is particularly important in this context. It would be useful to compare what the BdB in the base system has to offer compared to sqlite - because comparing to what the most recent versions of bdb in the ports can do has a different bearing on the question. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
On Mar 26, 2008, at 7:43 AM, Ivan Voras wrote: Pav Lucistnik wrote: Solution is to use tools that are available in our base system. SQLite is not. Yet :) Compared to some of the (huge) things that are maintained in the base, maintaining sqlite would be trivial :) But it's not as if the question is "sqlite or bust" - as OP noted, restructuring the system a bit and using better algorithms could fix many problems. The thing is - using something like sqlite is (at least for people used to SQL) much easier than rolling your own file system database (including locking and atomic ops). We're rehashing the discussion made last year around June - July. We came to the conclusion that BDB should be used, as no other DB backend / API exists in the base system (currently), and porting SQLLite (while nice) appeared to be non-trivial to port and got a lot of unhappy responses from folks. -Garrett ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
Pav Lucistnik wrote: > Solution is to use tools that are available in our base system. SQLite is not. Yet :) Compared to some of the (huge) things that are maintained in the base, maintaining sqlite would be trivial :) But it's not as if the question is "sqlite or bust" - as OP noted, restructuring the system a bit and using better algorithms could fix many problems. The thing is - using something like sqlite is (at least for people used to SQL) much easier than rolling your own file system database (including locking and atomic ops). signature.asc Description: OpenPGP digital signature
Re: ports system woes
On Wed, 2008-03-26 at 15:00 +0100, Michel Talon wrote: > Frank Mayhar wrote: > > > Portupgrade is, in fact, my preferred application, although > > lately I've had to move to portmaster just because of the O(n^2) > > inefficiencies. It doesn't need to be replaced, it needs to be fixed, > > I have rarely seen such a wonderful contradiction between the two halves > of the same sentence. There's no "contradiction," Michel. "Fixing" isn't the same thing as "rewriting" and despite the fact that I'm using portmaster at the moment, portupgrade _is_ my preferred application. > > Oh, please. The FreeBSD ports system works as well. Instead of > No, it doesn't or it does barely. Do you even _use_ it? It's working fine for me, modulo the few problems that have been caused solely by the increase in size of the collection. > > complaining, why don't you actually do someting about it? Code speaks > > louder than mere words. > We have learnt recently that many people have brought code to the table, > including myself, You mean your python code? What does it do that's useful? I ran it and it gives me a lot of output that's just not very interesting or useful and doesn't offer to correct anything. And what "many people?" When code is offered, I've seen it generally accepted, if it's complete. If you think the ports system should be rewritten, rewrite it. If it's better than the existing solution, I'm sure the world will beat a path to your door. Otherwise, well, them's the breaks. > only to be scorned by people of your sort. I have yet > to see people like you contributing any testing or discussion when code > is offered, all you are good for is parrotting "Code speaks louder than > mere words". I'm not the one who is complaining that the ports system is "broken," Michel. -- Frank Mayhar [EMAIL PROTECTED] http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ http://www.zazzle.com/fmayhar* ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
On Wed, 2008-03-26 at 14:59 +0100, Pav Lucistnik wrote: > On Wed, 26 Mar 2008 14:18:00 +0100, Michel Talon wrote > > > In fact last year i wrote a python script which reads all the > > /var/db/pkg/+CONTENTS files, and fixes all the +REQUIRED_BY files, > > assuming they are corrupted. Moreover it follows the MOVED file. > > So you basically reimplemented pkgdb -F in python? No. I'm not sure what he did implement, but it's not "pkgdb -F". > > As far as i remember this program > > runs in a few *seconds* certainly not minutes like it is said here > > Mind that the original poster is using a very low-spec notebook with next to > none RAM. That having been said, O(n^2) algorithms are generally not a good idea. > > solution is to use sqlite and not some half-assed solution like a > > Berkeley database, > > Solution is to use tools that are available in our base system. SQLite is not. Indeed. -- Frank Mayhar [EMAIL PROTECTED] http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ http://www.zazzle.com/fmayhar* ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
Frank Mayhar wrote: > Portupgrade is, in fact, my preferred application, although > lately I've had to move to portmaster just because of the O(n^2) > inefficiencies. It doesn't need to be replaced, it needs to be fixed, I have rarely seen such a wonderful contradiction between the two halves of the same sentence. > Oh, please. The FreeBSD ports system works as well. Instead of No, it doesn't or it does barely. > complaining, why don't you actually do someting about it? Code speaks > louder than mere words. We have learnt recently that many people have brought code to the table, including myself, only to be scorned by people of your sort. I have yet to see people like you contributing any testing or discussion when code is offered, all you are good for is parrotting "Code speaks louder than mere words". -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
On Wed, 26 Mar 2008 14:18:00 +0100, Michel Talon wrote > In fact last year i wrote a python script which reads all the > /var/db/pkg/+CONTENTS files, and fixes all the +REQUIRED_BY files, > assuming they are corrupted. Moreover it follows the MOVED file. So you basically reimplemented pkgdb -F in python? > As far as i remember this program > runs in a few *seconds* certainly not minutes like it is said here Mind that the original poster is using a very low-spec notebook with next to none RAM. > solution is to use sqlite and not some half-assed solution like a > Berkeley database, Solution is to use tools that are available in our base system. SQLite is not. -- Pav Lucistnik <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
On Wed, 2008-03-26 at 14:18 +0100, Michel Talon wrote: > More generally i disagree with Kris Kennaway idea that all is required > is to polish the existing tools. They are so obviously broken from all > points of view, particularly portupgrade, that only a complete rewrite > can do any good. Um, while I agree with part of what you said, this is just plain not true. The existing tools can easily be fixed as soon as someone takes the time to actually do it; a rewrite is neither necessary nor desirable. Portupgrade is, in fact, my preferred application, although lately I've had to move to portmaster just because of the O(n^2) inefficiencies. It doesn't need to be replaced, it needs to be fixed, and Pav's suggested fix is one place to start. > Needless to say, this cannot please those FreeBSD ports > afficionados who are convinced that their toy is the best in the world. > Let me recall that *one* person has completely rewritten the ports system > for OpenBSD (Marc Espie), including the pkg* tools and all the Makefile > scripts, and now it works. Oh, please. The FreeBSD ports system works as well. Instead of complaining, why don't you actually do someting about it? Code speaks louder than mere words. -- Frank Mayhar [EMAIL PROTECTED] http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ http://www.zazzle.com/fmayhar* ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
Pav Lucistnik wrote: > Quick solution would be to gather all depnames for the deleted package, > and then do a single pass over /var/db/pkg entries looking for origins. > > Ultimate solution would be to implement a database which would > concentrate origins for all packages with linear lookup time. In fact last year i wrote a python script which reads all the /var/db/pkg/+CONTENTS files, and fixes all the +REQUIRED_BY files, assuming they are corrupted. Moreover it follows the MOVED file. Optionnally it decomposes the set of ports in connected components, and builds graphviz graphs for the dependencies. Of course the leaf packages are written down. As far as i remember this program runs in a few *seconds* certainly not minutes like it is said here for the pkg_delete stuff. You can find it here: http://www.lpthe.jussieu.fr/~talon/pkg_check.py This being said, i agree with you, that for various reasons, the package system needs to get rid of the extremely bad idea of abusing the filesystem as a database, and use a true database for doing its job. This would allow O(1) searches, as you are saying, and would allow to perform appropriate locking for supporting parallel builds. We had this discussion last year, and i am even more convinced that the good solution is to use sqlite and not some half-assed solution like a Berkeley database, if only because using a sql database will lead naturally to a more structured solution, and not a pile of blobs, and also because sqlite is a stable software. More generally i disagree with Kris Kennaway idea that all is required is to polish the existing tools. They are so obviously broken from all points of view, particularly portupgrade, that only a complete rewrite can do any good. Needless to say, this cannot please those FreeBSD ports afficionados who are convinced that their toy is the best in the world. Let me recall that *one* person has completely rewritten the ports system for OpenBSD (Marc Espie), including the pkg* tools and all the Makefile scripts, and now it works. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ports system woes
Hi, I have read your mail on hackers@ (I'm not subscribed) with great interest. We're certainly interested in any optimizations for the pkg_install suite. The pkg database in /var/db/pkg stores two-way dependency chains. Each port lists all it's dependencies in +CONTENTS file, and all ports that depend on it in +REQUIRED_BY. When you delete package, all dependencies of deleted package are iterated and the name of deleted package is removed from dependency's +REQUIRED_BY file. That's what undepend() do. Quick solution would be to gather all depnames for the deleted package, and then do a single pass over /var/db/pkg entries looking for origins. Ultimate solution would be to implement a database which would concentrate origins for all packages with linear lookup time. The OpenSSL thing I assume is only relevant for people who happen to have OpenSSL installed from ports. For that, it could be solved by spamming the required value into /etc/make.conf, similar what perl ports do. But that really is up to the openssl port maintainer ([EMAIL PROTECTED]). -- Pav Lucistnik <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> If God is perfect, why did He create discontinuous functions? signature.asc Description: Toto je digitálně podepsaná část zprávy
Fwd: ports system woes
There is a newly-spawned discussion on -hackers about the inefficiencies the existing pkg_* utiliies (e.g. pkg_delete taking minutes to operate, etc.): http://lists.freebsd.org/pipermail/freebsd-hackers/2008-March/024057.html Those who wish to participate should probably be people who are familiar with both the ports framework and actual C code; IMHO it should not be a "let's talk about features I want" thread. Just bringing this to other technical users' attention who may feel inclined to participate. Stephen, I'm CC'ing you because I know you've put in great efforts fixing said inefficiencies in the past. :-) -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "[EMAIL PROTECTED]"
