On 5/20/06, Brannon King <[EMAIL PROTECTED]> wrote:
The benefits I'm trying to get out of sqlite are the data queries. I collect a large, sparse 2D array from hardware. The hardware device is giving me a few GB of data data at 200MB/s. Future hardware versions will be four times that fast and give me terabytes of data. After I have the data, I then have to go through and make calculations on sub-boxes of that data.
Have you thought about constructing a data warehouse for this? Use something close to the metal (hash + binary file I/O) to store the initial data. Then write something to load what data you need into a database. You can then do your queries against that. I think your overall bottle neck is indexing, and indexing 800MB/s sustained will take big iron or a beowulf cluster.
