Dear all, I have a database with about 5000 tables each with more than 1 million records. I needed to get some summary statistics of each table but find that it will take days to run 'SELECT count(*) FROM table_XX' (XX=1,...,5000) sequentially. I therefore created 10 threads, each having its own database connection (sqlite3.connect() from python). To my dismay, it appears that only one thread was allowed to access the database at any time so the overall speed was not improved. I further created 10 processes, each running one query (e.g. 'sqlite3 file:mydatabase?mode=ro "SELECT count(*) FROM table_XX"'), but I still could see any improvement in performance...
Because each thread/process took only 10M RAM and about 1% CPU, staring 10 threads/processes should not be a problem at all. Before I investigate further (e.g. programming error, bottleneck of disk I/O), can anyone tell me if I am going the right direction? I mean, is it possible, in theory, to read a sqlite database from multiple threads/processes each with performance comparable to a single thread/process? I understand writing to a database from multiple threads can be troublesome but I only need to read from it. Many thanks in advance, Bo _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
