Monday, May 22, 2006, 15:17:21, Jay Sprenkle wrote: > On 5/22/06, Dennis Jenkins <[EMAIL PROTECTED]> wrote: >> Brannon King wrote: >> > The benefits I'm trying to get out of sqlite are the data queries. I >> > collect a large, sparse 2D array from hardware. The hardware device is >> > giving me a few GB of data data at 200MB/s. Future hardware versions >> > will be four times that fast and give me terabytes of data. After I >> > have the data, I then have to go through and make calculations on >> > sub-boxes of that data. (I'll post some more about that in a different >> > response.) I was trying to avoid coding my own >> > sparce-matrix-file-stream-mess that I would have to do if I didn't >> > have a nice DB engine. I think sqlite will work. I think it will be >> > fast enough. I'll have some nice RAID controllers on the production >> > machines with 48-256MB caches. >> >> Hello Brannon, >> >> I am simply curious. This sounds like an amazing engineering >> challenge. If it is not a secret, can you describe what this data >> represents and how it will be used?
> Me too! Me too too :) _Personally_ I think, this sounds like a task not quite fitting in sqlites (probably any 'standard' databases) realm. This is a bit off-topic in this group, but because you mention sub-boxes - did you ever look into more specialized file-formats like HDF5: http://hdf.ncsa.uiuc.edu/HDF5/ with support for certain (of course limited) queries ? Micha --
