Richard Hipp wrote: > So you are creating a 27.5 GB database in under a half hour? > > What is your database page size?
Yes, to be exact, a 34.24GB database from (34GB+34GB) FASTQ files in that time. I did not use any pragmas for swiching SQLite page size, so it should be the default (probably 4096?). > You might try programming a virtual table to read you FASTQ files, then > transfer content from the virtual table into your real table using: > > INSERT INTO realtable SELECT * FROM virtualtable; > > Information on virtual tables is at > http://www.sqlite.org/vtab.html > and you > can find example code in some of the extensions at > > http://www.sqlite.org/src/tree?ci=trunk&name=ext/misc > and in some of the > test modules at > http://www.sqlite.org/src/tree?ci=trunk&name=src&re=test_ It might be possible, but virtual tables live in memory (I am developing on 16GB), while my source files over 30GB each (I originally read the entire file in a single operation into a string, didn’t work with the 30GB files, which is why I switched to boost::mmap), so I don’t think it would work on my platform. However, since the application is intended to be used on a computing cluster (a generic node in our cluster would have 72GB of RAM, but I could request 384GB “fat” nodes), it could work. However, I was working with 2 faculty with conflicting views on the matter (the biochemist wants everything to work on low-powered systems, the computer scientist is sure that everything would work faster if I got the algorithm correct). Kevin. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
