On 14 Jul 2012, at 2:31pm, Udi Karni <uka...@gmail.com> wrote: > (4) Get enough RAM to contain all the data and swap
This may be the best solution to the problem but it's disappointing. You really shouldn't need to do this. Computers and caching algorithms should be doing better to help you. > In light of that - is it completely out of line to consider "light > parallelization" for Sqlite? Even just for the simplest SELECTs (SELECT > COUNT (*) WHERE AGE = NNN) ? Have Sqlite spawn 2 processes - 1 counting > the 1st half of the blocks of a table - the second counting the 2nd half? > Limited only to reads where there are no locking issues? Even capped at 2 > threads? Only for tables greater than xGB so as not to bother with the tiny > ones? Would that introduce bloat in the code and detract from Sqlite's > original mission? Parallelization is of great advantage when the problem is processing: maths, pattern-searching, and processing of lots of data already in memory. But SQLite is very efficient at doing this, with a very simple very searchable file format. I think your bottleneck here isn't processing, it's I/O, just as you put in the 'Subject' header. I bet your limitation is in your bus width, throughput, or low-level file handling. If you implement parallelization the way you describe, the result will just be the two commands constantly fighting over access to your datastore -- back to bandwidth and throughput again. Just to give you an idea, in normal setups the normal bottleneck for SQLite speed is the rotational speed of a hard disk. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
