On 2 Apr 2014, at 4:58am, Kevin Xu <accol...@gmail.com> wrote: > The app works by reading in the FASTQ file (through Boost::memory_mapped > regions), running the parser through it, binding the variables using > sqlite3_bind_xxx, and inserting them in a prepared statement wrapped within > BEGIN...END statements.
Just wanted to check your phrasing. You process a while file into INSERT commands, then put BEGIN before the first one, and END after the last one, thus making the whole file a single transaction ? If so, then should make SQLite do things about as quickly as it can. Though I think a huge transaction might actually slow things down if your database is in WAL mode. > my professor was adamant that he managed to insert 3 million rows in 7 > seconds (which works out to over 420K inserts per second) though he could > not find his code that did it or records of his insertions. If he did, and that does seem very fast, I would guess he used some PRAGMAs to speed things up, or did it on a virtual machine where the whole operation was done in memory and flushed to disk later. If you're running on a Virtual Machine then you have lost ACID anyway, so you might as well sacrifice all durability using PRAGMAs like PRAGMA synchronous = OFF If your program needs to do other things after the huge inserts are finished, close the database connection and open a new one to make sure changes are written to disk and the PRAGMAs are reset. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
