Hi Howard, > I just looked, sophia is nothing special. See these microbench results. > http://pastebin.com/cFK1JsCN > > LMDB's codebase is still smaller and faster. Nothing else touches LMDB's > read > speed.
This is micro benchmark from sophia author compare with lmdb. http://sphia.org/benchmarks.html The benchmark is different in: 1. All read or write benchmark shouldn't be done in one transaction, but you do. 2. Between one benchmark to another (run read test just after a write one), try to drop all caches in-memory and open database, this can model situation when data can't be in-memory because database is too high to fit them in. 3. Because sophia is optimized for write, the benchmark use 3 million and 10 million record. Hope this is can be an input for your development. On 11/5/13, Raheel Gupta <raheel...@gmail.com> wrote: >>> SQLite4 still in development phase. It is not production ready. > > But isnt that the same thing as BDB or Kyoto i.e. a Key Value store ? > >>> If you have many core of processors [and big RAM], then I recommend > BDB Sql over Sqlite. > > I have large space and around 4GB of ram with Dual Cores to Quad Cores > processors meant only for storage. > >>> If you can choose DBMS, other than SQLite, try to use DB that have > storage engine optimized for write, for example LSM (hypertable), > Fractal Tree (tokudb engine for mysql). > > I would be interested in a embedded DB which can give good performance i.e. > write fast with indexing and read fast as well. > > > On Mon, Nov 4, 2013 at 6:42 PM, Aris Setyawan <aris.s...@gmail.com> wrote: > >> > Will SQLite4 be a better solution for me then ? >> >> SQLite4 still in development phase. It is not production ready. >> >> > Also @aris do you recommend BDB over Sqlite for 1-10 Billion records ? >> >> If you have many core of processors [and big RAM], then I recommend >> BDB Sql over Sqlite. Because you can have many processes or threads to >> write to a database concurrently. Because it use row or page level >> locking. >> >> If you can choose DBMS, other than SQLite, try to use DB that have >> storage engine optimized for write, for example LSM (hypertable), >> Fractal Tree (tokudb engine for mysql). >> >> On 11/4/13, Raheel Gupta <raheel...@gmail.com> wrote: >> > Will SQLite4 be a better solution for me then ? >> > >> > Also @aris do you recommend BDB over Sqlite for 1-10 Billion records ? >> > >> > >> > On Mon, Nov 4, 2013 at 8:03 AM, Aris Setyawan <aris.s...@gmail.com> >> wrote: >> > >> >> > I just looked, sophia is nothing special. See these microbench >> results. >> >> > http://pastebin.com/cFK1JsCN >> >> > >> >> > LMDB's codebase is still smaller and faster. Nothing else touches >> >> > LMDB's >> >> > read >> >> > speed. >> >> >> >> Focus to the write number. >> >> >> >> You are using SSD or HDD? >> >> >> >> On 11/4/13, Howard Chu <h...@symas.com> wrote: >> >> > Aris Setyawan wrote: >> >> >>> SQLightning replaces the SQLite backend with Symas' LMDB, which >> >> >>> also >> >> >>> uses >> >> >>> MVCC >> >> >>> and thus supports high concurrency. It is also many times faster >> than >> >> >>> BerkeleyDB and vanilla SQLite. >> >> >> >> >> >> Your MVCC is different compared to InnoDB or BDB locking. Every one >> >> >> should carefully read each DB's doc, then test it before decide to >> use >> >> >> it. >> >> > >> >> > Yes, it's different. In LMDB writers never block readers and readers >> >> never >> >> > block writers. The original poster was complaining about SELECT >> >> > taking >> >> > a >> >> > long >> >> > time and preventing other threads from making progress. That problem >> >> doesn't >> >> > >> >> > exist in LMDB. BDB locking *might* be able to avoid this in many >> cases, >> >> if >> >> > there are no hotspots, but is prone to deadlocks which require the >> >> calling >> >> > application to retry failed requests. >> >> > >> >> >> LMDB is storage engine optimized for read. Why you don't optimize >> it's >> >> >> write using cache oblivious data structure, for example LSM tree or >> >> >> create new, like in sophia (sphia.org) key value DB? >> >> > >> >> > I just looked, sophia is nothing special. See these microbench >> results. >> >> > http://pastebin.com/cFK1JsCN >> >> > >> >> > LMDB's codebase is still smaller and faster. Nothing else touches >> >> > LMDB's >> >> > read >> >> > speed. >> >> > >> >> > -- >> >> > -- Howard Chu >> >> > CTO, Symas Corp. http://www.symas.com >> >> > Director, Highland Sun http://highlandsun.com/hyc/ >> >> > Chief Architect, OpenLDAP http://www.openldap.org/project/ >> >> > _______________________________________________ >> >> > sqlite-users mailing list >> >> > sqlite-users@sqlite.org >> >> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> > >> >> _______________________________________________ >> >> sqlite-users mailing list >> >> sqlite-users@sqlite.org >> >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> >> >> > _______________________________________________ >> > sqlite-users mailing list >> > sqlite-users@sqlite.org >> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
