On Fri, 27 Aug 2010 18:19:53 +0400 denisgolovan <[email protected]> wrote: > 27.08.10, 14:38, "Ciaran McCreesh" <[email protected]>: > > The search index becomes invalid after you change *anything*. If > > you sync, it's invalid. If you install, upgrade, reinstall or > > uninstall something, it's invalid. If you change any of your > > configuration files, it's invalid. > > Well, I thought about global index. > I mean that is under paludis full control. > Imho, doing installs/uninstalls from paludis/portage in the same time > is not the right practice anyway.
Unfortunately, we can't rely upon users being sensible... > Not sure about difficulties with configuration files. > Their size/format is not that big/complex compared to > ebuilds/eclasses. But I am in no way an expert here. In order to know whether a configuration file change *might* have invalidated the index, we'd have to store a full copy of all of the config files beforehand, and then compare it every time. There's no point in doing so. > > There's no performance gain to be had from incremental updates > > after a sync. After a sync, to build an index, every bit of > > metadata has to be checked to see whether it's changed, which is > > where the time goes (and the cost of removing a subset of invalid > > things and reinserting new values into an SQLite table is pretty > > much the same as just rewriting the entire thing from scratch). > > Maybe some kind of check-sums for ebuilds + eclasses can be useful > here? Metadata is already validated based upon mtimes. The slow part is going through and checking the mtimes on 20,000+ files and comparing them with the stored values in another 20,000+ files and seeing which ones are no longer valid. (And we can't do anything about that -- the format there is controlled by Gentoo, and it's generated centrally.) > I have ~20 Gentoo repositories with 32000 ebuild in total and it took > me about two hours to build the index. I can hardly imagine any > number of queries for justifying such index rebuilds (besides some > centralized search). That sounds highly suspicious. Are you using a CVS checkout of the main Gentoo repository or something? If you get ::gentoo via rsync, it comes with ebuild metadata pregenerated, so building the index should only take a couple of minutes for that. And you shouldn't have *that* many packages in overlays... > Again, maybe it worth doing some "light" information index storage? > I mean putting only more or less static information in db. No point. If you're putting enough information in the db to make searches fast, you have to have the db up to date with metadata. The search index is just designed to make repeated searches on the same data fast. If you want anything more than that, you either have to sacrifice the 'fast' part (in which case there's no point having an index at all), or give incorrect answers. -- Ciaran McCreesh
signature.asc
Description: PGP signature
_______________________________________________ paludis-user mailing list [email protected] http://lists.pioto.org/mailman/listinfo/paludis-user
