[sqlite] MMAP performance with databases over 2GB

2015-08-08 Thread Howard Chu
Roger Binns wrote: > It would also be really nice if there wasn't a 2GB mmap limit on 64 > bit machines. The database would fit in my RAM around 4 times, and in > the address space more times than there are grains of sand! Yea I > know this isn't very Lite ... SQLightning has no such limit...

[sqlite] MMAP performance with databases over 2GB

2015-08-07 Thread Dan Kennedy
On 08/07/2015 12:35 AM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 08/06/2015 09:27 AM, Dan Kennedy wrote: >>> Is it using more CPU cycles in mmap mode or just taking longer? >>> If the former, does [time] attribute them to "user" or "system"? > It is taking

[sqlite] MMAP performance with databases over 2GB

2015-08-07 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/07/2015 02:55 AM, Dan Kennedy wrote: >> http://www.sqlite.org/src/info/3a82c8e6cb7227fe > >> Does that improve performance any in your case? I tested 3a82c8e6 (fix above) against e596a6b6 (previous changeset), in both cases starting with an

[sqlite] MMAP performance with databases over 2GB

2015-08-07 Thread Dan Kennedy
On 08/06/2015 09:53 PM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > In my testing I am finding that using SQLite's mmap support with > larger databases (final db size is ~10GB) to create the database (WAL > mode) is considerably (about three times) slower than no mmap.

[sqlite] MMAP performance with databases over 2GB

2015-08-06 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/06/2015 09:27 AM, Dan Kennedy wrote: >> Is it using more CPU cycles in mmap mode or just taking longer? >> If the former, does [time] attribute them to "user" or "system"? It is taking longer. I have 3 XML dumps which I turn into JSON

[sqlite] MMAP performance with databases over 2GB

2015-08-06 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 In my testing I am finding that using SQLite's mmap support with larger databases (final db size is ~10GB) to create the database (WAL mode) is considerably (about three times) slower than no mmap. The default max mmap limit is 2GB (sadly). Has