On Thu, 21 Dec 2006, seth vidal wrote:
On Thu, 2006-12-21 at 09:30 +0200, Panu Matilainen wrote:
On Thu, 21 Dec 2006, seth vidal wrote:
On Wed, 2006-12-20 at 10:47 +0200, Panu Matilainen wrote:
I wrote a quick-n-dirty sqlite "backend" for apt to see how things would
look like for another implementation (with totally different usage
patterns) of repodata.
The initial version performs so badly you wouldn't believe it :D
On 2.4GHz AMD64 the creation of dependency cache (which involves walking
over all packages and recording the dependency data to it's the memory
mapped dependency cache) takes over 3 MINUTES (!) for just FC6 core
data. With xml repodata, that's ~6.5s operation on this system, fully
reading filelists.xml info as well, which the sqlite version doesn't do at
all at this point.
After a bit of investigation, the major bottleneck here is that the
provides, requires, conflicts and obsoletes tables don't have index on
pkgKey. After creating those indexes, it's back to ~6.5s even with the
naive initial implementation.
I haven't done any timings on how those indexes would affect yum's usage
patterns, probably not *that* dramatic but it might be something to look
at.
I'm sure that better indexing would help some lookups rather a lot.
Curiously enough, the pkgKey indexes on prco data don't affect yum at all,
even though one would think eg returnPrco() would benefit a great deal.
odd, I'd thought it would have benefitted, too.
Indeed. It's so "obvious" that I feel I must be overlooked something but I
just don't see what.
out of curiosity - how much does the size of the sqlite file go up when
the indexes are added?
For FC-development primary db:
-rw-r--r-- 1 pmatilai pmatilai 5074944 Dec 21 09:17 primary.xml.gz.sqlite.orig
-rw-r--r-- 1 pmatilai pmatilai 5650432 Dec 21 09:18 primary.xml.gz.sqlite
So it's certainly a non-trivial amount of metadata on metadata which yum
doesn't even seem to care about :)
How well they do compare, compressed? Do the indexes compress better
than we might think? :)
Heh, I forgot about compression completely :) That way things seem a whole
lot better:
1619533 Dec 21 09:17 primary.xml.gz.sqlite.orig.gz
1741906 Dec 21 09:18 primary.xml.gz.sqlite.gz
1223076 Dec 21 09:17 primary.xml.gz.sqlite.orig.bz2
1299333 Dec 21 09:18 primary.xml.gz.sqlite.bz2
- Panu -
_______________________________________________
Yum-devel mailing list
[email protected]
https://lists.dulug.duke.edu/mailman/listinfo/yum-devel
