27.08.10, 18:33, "Ciaran McCreesh" : > On Fri, 27 Aug 2010 18:19:53 +0400 > denisgolovan wrote: > > 27.08.10, 14:38, "Ciaran McCreesh" : > > > The search index becomes invalid after you change *anything*. If > > > you sync, it's invalid. If you install, upgrade, reinstall or > > > uninstall something, it's invalid. If you change any of your > > > configuration files, it's invalid. > > > > Well, I thought about global index. > > I mean that is under paludis full control. > > Imho, doing installs/uninstalls from paludis/portage in the same time > > is not the right practice anyway. > > Unfortunately, we can't rely upon users being sensible...
Too bad :) Does Exherbo have the same problem? > > > There's no performance gain to be had from incremental updates > > > after a sync. After a sync, to build an index, every bit of > > > metadata has to be checked to see whether it's changed, which is > > > where the time goes (and the cost of removing a subset of invalid > > > things and reinserting new values into an SQLite table is pretty > > > much the same as just rewriting the entire thing from scratch). > > > > Maybe some kind of check-sums for ebuilds + eclasses can be useful > > here? > > Metadata is already validated based upon mtimes. The slow part is going > through and checking the mtimes on 20,000+ files and comparing them > with the stored values in another 20,000+ files and seeing which ones > are no longer valid. > > (And we can't do anything about that -- the format there is controlled > by Gentoo, and it's generated centrally.) Hm, I see now. > > I have ~20 Gentoo repositories with 32000 ebuild in total and it took > > me about two hours to build the index. I can hardly imagine any > > number of queries for justifying such index rebuilds (besides some > > centralized search). > > That sounds highly suspicious. Are you using a CVS checkout of the main > Gentoo repository or something? If you get ::gentoo via rsync, it comes > with ebuild metadata pregenerated, so building the index should only > take a couple of minutes for that. And you shouldn't have *that* many > packages in overlays... Hm, my fault. Pregenerated cache for gentoo, zugaina repositories was not used at all. Now it is 4.5 minutes (cold file cache), even though zugaina has a lot of "stale cache" warnings emitted. Nice improvement :) Thanks for the idea. > > Again, maybe it worth doing some "light" information index storage? > > I mean putting only more or less static information in db. > > No point. If you're putting enough information in the db to make > searches fast, you have to have the db up to date with metadata. > > The search index is just designed to make repeated searches on the same > data fast. If you want anything more than that, you either have to > sacrifice the 'fast' part (in which case there's no point having an > index at all), or give incorrect answers. Surely I can miss something, but in default configuration, doing pretty standart query like: $ cave search --index ~/cave-search-index gcc .. I get information for package name, all version available in repositories, descriptions, homepage, installed versions and masks. First four kinds are static between syncs, aren't they? And for them it's enough to have static cache and fast search. If a user wants some more, cave joins (in db relational sense) this information at runtime. Seems like a valid trade-off. Please criticize. _______________________________________________ paludis-user mailing list [email protected] http://lists.pioto.org/mailman/listinfo/paludis-user
