On Thu, 2006-12-21 at 09:30 +0200, Panu Matilainen wrote: > On Thu, 21 Dec 2006, seth vidal wrote: > > > On Wed, 2006-12-20 at 10:47 +0200, Panu Matilainen wrote: > > > >> I wrote a quick-n-dirty sqlite "backend" for apt to see how things would > >> look like for another implementation (with totally different usage > >> patterns) of repodata. > >> > >> The initial version performs so badly you wouldn't believe it :D > >> > >> On 2.4GHz AMD64 the creation of dependency cache (which involves walking > >> over all packages and recording the dependency data to it's the memory > >> mapped dependency cache) takes over 3 MINUTES (!) for just FC6 core > >> data. With xml repodata, that's ~6.5s operation on this system, fully > >> reading filelists.xml info as well, which the sqlite version doesn't do at > >> all at this point. > >> > >> After a bit of investigation, the major bottleneck here is that the > >> provides, requires, conflicts and obsoletes tables don't have index on > >> pkgKey. After creating those indexes, it's back to ~6.5s even with the > >> naive initial implementation. > >> > >> I haven't done any timings on how those indexes would affect yum's usage > >> patterns, probably not *that* dramatic but it might be something to look > >> at. > > > > I'm sure that better indexing would help some lookups rather a lot. > > Curiously enough, the pkgKey indexes on prco data don't affect yum at all, > even though one would think eg returnPrco() would benefit a great deal.
odd, I'd thought it would have benefitted, too. > > > out of curiosity - how much does the size of the sqlite file go up when > > the indexes are added? > > For FC-development primary db: > -rw-r--r-- 1 pmatilai pmatilai 5074944 Dec 21 09:17 primary.xml.gz.sqlite.orig > -rw-r--r-- 1 pmatilai pmatilai 5650432 Dec 21 09:18 primary.xml.gz.sqlite > > So it's certainly a non-trivial amount of metadata on metadata which yum > doesn't even seem to care about :) How well they do compare, compressed? Do the indexes compress better than we might think? :) -sv _______________________________________________ Yum-devel mailing list [email protected] https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
