My first thought would be to check the amount of memory being used by your many connections. Each connection will consume a non-trivial amount of resources (page cache, file handles, OS file cache, etc.) It's certainly plausible that your overall system performance is reduced as you run out of physical memory (or other system resources). As such, I'd double check your free / available memory as you open more and more connections-- see if there is a significant impact.
On a side note, trying to manage 100K or more separate databases sounds excessive. Doing so somewhat defeats the purpose of a nice relational database. If you properly index your tables, I would think you could still achieve similar / reasonable performance, even after combining the many smaller databases into fewer larger ones. [Just my 2 cents.] On 9/2/11, Terry Cumaranatunge <cumar...@gmail.com> wrote: > Hello, > > We have an application that creates many small databases (over 100K) to be > able to control the size of the database and provide more deterministic > performance. At process startup, it opens many of these databases in a loop > to keep them open for faster transaction response times. The behavior we are > noticing is that the it takes progressively a longer time for each > sqlite3_open_v2 to complete as the number of databases kept > opened increases. These are some of the measurements: > > 5000 DBs = 3 secs > 10000 DBs = 11 secs > 15000 DBs = 35 secs > 20000 DBs = 75 secs > > Many processes can concurrently open 5000 db's at the same time and it takes > about the same time as a single process doing the work. So, it doesn't > appear to be related to the OS related issue with the number of opens. > > The open is done as: > sqlite3_open_v2(dbName, db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_NOMUTEX, > NULL) > > We are using RHEL 6 with SQLite 3.7.7.1. Each database was created with > journaling mode set to WAL. > Does anyone know why the performance degrades this way and what can be done > to resolve the problem? > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
