On Sat, 20 May 2006, Brannon King wrote:
John Stanton wrote:
You don't seem to need a data manipulation system like Sqlite, more a form
of high volume storage. Do you really need elaborate SQL, journalling,
ROLLBACK and assured disk storage?
Di you consider some form of hashed storage, perhaps linear hashing, to
build a compact and high performance associative array for your sparsely
keyed data.
Do you really need the overhead of B-trees is you are just storing a sparse
array?
JS
I don't need journaling or rollback. I'd love a way to shut them off. But
elaborate SQL, that sure is handy. I'm not just storing, I'm viewing stored,
compressed data. I definitely need some way of querying a sparse matrix data
that is larger than my DRAM. Sqlite sure seems like the quickest route to a
workable product for that to happen. It has all the streaming/caching built
in. Because of that, I assume it is faster than random file access. It
supports complex data queries and indexes, both things I would need anyway.
In the world of programming, I think many will agree you should get a working
product, then make it faster. I'm just trying to get the most speed out of
the easiest tool. If I need to rewrite the file storage for the next version,
we can consider the cost to benefit for that separately.
Is that 800MB/s sustained? Is it constant, 24/7? Or is it bursty data?
If it's bursty, you could buffer the inserts in an append only log.
Insertion to the SQL(ite) database can be done asynchronously. Writing to
unstructured, append only log files can be done at pretty much disk I/O
speed.
If it's not bursty, but sustained, then I fear SQLite or any other
database will not be able to match that rate. You'll probably run out of
CPU before running out of disk IO.
Christian
PS. On a side note, is it wise still buying SGI kit?
