Thank you all for your advice. I use the standard Shell compiled for 64-bit Windows - so it's single threaded - and the only way to multi-thread is to run 2 concurrent processes on 2 concurrent Sqlite DBs - assuming there would be no conflict with the master tables - since they are read only - I'll need to test that.
Otherwise - seems like the options are - (1) Windows compile optimizations (2) OS optimizations (3) Try Linux which is typically zippier (4) Get enough RAM to contain all the data and swap I completely understand why Sqlite needs to be "lite" - because it's meant for small devices running in RAM. It's just so clever - nothing to install - no threads, etc. - it's tempting to use on the desktop for conventional DBs - where you really need a parallel engine if you want to cut through large amounts of data quickly. In light of that - is it completely out of line to consider "light parallelization" for Sqlite? Even just for the simplest SELECTs (SELECT COUNT (*) WHERE AGE = NNN) ? Have Sqlite spawn 2 processes - 1 counting the 1st half of the blocks of a table - the second counting the 2nd half? Limited only to reads where there are no locking issues? Even capped at 2 threads? Only for tables greater than xGB so as not to bother with the tiny ones? Would that introduce bloat in the code and detract from Sqlite's original mission? Thanks. On Sat, Jul 14, 2012 at 5:58 AM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 14 Jul 2012, at 5:40am, Udi Karni <uka...@gmail.com> wrote: > > > It very easily handles billion row/100GB tables - multi-table joins, > etc. - > > it just chugs for a while because it's single threaded, and will gobble > up > > memory and swap - but it gets the job done. > > ... though you can, of course, do your SQLite call in a second thread and > proceed as normal on your main thread. Still single-threaded but it won't > block. I've noticed a lot of web pages which build the page, then fill in > the contents of a table later. > > > It's quite amazing for a DB engine that's not even 1 MB. > > It's kept small partly because it's used in tiny single-core embedded > systems: phone handsets, handheld control devices, and machine controllers. > I even found a SQLite header in the firmware from my TV recorder. It has > to run under tiny amounts of memory on simple hardware with little or no > caching. What's interesting is that somehow a SQL engine designed for > hand-sized devices is so good on standard desktop computers that many > people use it. > > > While it's happiest when it can do all the work in memory versus disk - > it > > reads fairly quickly from disk the first time around - but not > subsequently > > - which is why I posed the question. > > This characteristic is something to do with your hardware or OS, not > something inherent in SQLite. I bet if you tried the same thing on a Mac > or Linux you wouldn't get the same behaviour. > > I seem to remember that Windows caches files with certain extensions > specially, but I can't find any elucidation on the web. > > Simon. > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
