On 29 Mar 2013, at 7:17pm, Jeff Archer <jsarc...@nanotronicsimaging.com> wrote:
> This use case is from an image processing application. I have a large > amount of intermediate data (way exceeds physical memory on my 24GB > machine). So, I need to store it temporarily on disk until getting to next > phase of processing. I am planning to use a large SSD dedicated to holding > this temporary data. I do not need any recoverability in case of hardware, > power or other failure. Each item to be stored is 9 DWORDs, 4 doubles and > 2 variable sized BLOBS which are images. > > I could write directly to a file myself. That's what I thought: just blat it straight out to a text file in append mode. Organise it later. But if you don't want to do that ... > But I would need to provide some > minimal indexing, some amount of housekeeping to manage variable > sized BLOBS and some minimal synchronization so that multiple instances of > the same application could operate simultaneously on a single set of data. Locking between different simultaneous apps is going to kill you. Do you really need, at this stage, one app to access data written by different writers ? If not, have each app write to a different database. Then use a different app running on a different computer (or at least in a different process) to merge the data into one big file, either simultaneously or after data-collection is finished. > So, then I though that SQLite could manage these things nicely for me so > that I don't have to write and debug indexing and housekeeping code that > already exists in SQLite. > > So, question is: What is the way to get the fastest possible performance > from SQLite when I am willing to give up all recoverability guarantees? > Or, is it simple that I should just write directly to file myself? <http://www.sqlite.org/pragma.html> Journalling off and Synchronous off pragmas. And I note you're asking for 'fastest' whereas you really just want 'fast enough'. However I note that you seem to have >24Gig of data to write. At that size, the speed of SQLite isn't your limiting factor. Instead you have to worry about the speed of your storage medium. With 4K sectors, writing 24Gig of data means you're writing 3 million sectors. If you're writing to rotating hard disks that means you'll be waiting for 3 million sectors to be rotated into the right place. Even at 10,000 RPM that's a lot of waiting. Your biggest speed increase isn't going to come from clever programming, it's going to come from moving to SSD. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
