I tried the SQLite in-memory DB feature - but when it exceeded about 2GB - I got the "out of memory" message.
In my particular scenario - while the raw data being attached and read is hundreds of GB - the result sets are only a few GB. A 64-bit version of SQLite that could handle an in-memory DB of 5-10 GB would be quite useful and interesting to test. Unfortunately I am not versed enough in compiling my own version - so hopefully this would be offered one of these days. Whether a RAM-only version would truly run subtantially faster than an SSD-based DB - remains to be seen - but the <2GB experiments seem to show that in-memory is quite promising. On Thu, Mar 22, 2012 at 8:29 AM, Don V Nielsen <donvniel...@gmail.com>wrote: > There is a natural 5th extrapolation: > > 5) Could sqlite3 take advantage of multiple cpu's by parsing a single task > into one thread per cpu and segment data to be worked by each thread? Big > league stuff. But I don't think sqlite3 is meant to compete in that > market. It already exceeds expectations in its current market. > > dvn > > On Wed, Mar 21, 2012 at 6:57 PM, Udi Karni <uka...@gmail.com> wrote: > > > Frankly I don't know if a 64-bit version and Big RAM would make a > > difference and if so - up to what point. With SQLite being a single > process > > - assigned for the most part to a single CPU - even if everything was > done > > in RAM - there is a limit to what 1 CPU can do. > > > > I am just noticing anecdotally that SQlite uses cache and dealing with > > tables of a few hundred MB or less doesn't seem to generate IO. Also - > when > > there is IO - it often comes from the swap file (under Windows 7). > > > > So the questions are - > > > > (1) how much RAM is the point of diminishing returns on 32-bit > > (2) is there value to going 64-bit > > (3) if there was a 64-bit version - would it use more RAM more > effectively? > > (4) as a fallback - let's say the 32-bit version and 4GB are as good as > you > > can pretty much expect. Would getting a server with 4 CPUs and 16GB (a > > high-end home-version PC) - reasonably enable me to run 3-4 SQLite jobs > > concurrently? In other words - no great speed improvement per job - but > in > > aggregate more work could get done? > > > > Thanks ! > > > > On Wed, Mar 21, 2012 at 12:26 PM, Roger Binns <rog...@rogerbinns.com> > > wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA1 > > > > > > On 21/03/12 11:09, Black, Michael (IS) wrote: > > > > Cache is the primary (and obvious) thing I can think of. > > > > > > With a 32 bit compilation you'll be able to bump it up to about 2GB. > > > However by that point you will long have passed diminishing returns and > > > can just let the OS do its own caching. > > > > > > Roger > > > -----BEGIN PGP SIGNATURE----- > > > Version: GnuPG v1.4.11 (GNU/Linux) > > > > > > iEYEARECAAYFAk9qK2IACgkQmOOfHg372QQVdwCfbJTAzhCPR4ARPxhYHewLvvcT > > > 4lYAoI4QFXFfxILtsQGxVWm8BRM/mbIX > > > =e0aW > > > -----END PGP SIGNATURE----- > > > _______________________________________________ > > > sqlite-users mailing list > > > sqlite-users@sqlite.org > > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
