At 21:28 20/05/2006, you wrote:
The benefits I'm trying to get out of sqlite are the data queries. I
collect a large, sparse 2D array from hardware. The hardware device
is giving me a few GB of data data at 200MB/s. Future hardware
versions will be four times that fast and give me terabytes of data.
After I have the data, I then have to go through and make
calculations on sub-boxes of that data. (I'll post some more about
that in a different response.) I was trying to avoid coding my own
sparce-matrix-file-stream-mess that I would have to do if I didn't
have a nice DB engine. I think sqlite will work. I think it will be
fast enough. I'll have some nice RAID controllers on the production
machines with 48-256MB caches.
From experimentation, the "UNIQUE INDEX" command is only 20% slower
than the "INDEX" command. That doesn't make sense to me. I would
think the UNIQUE INDEX creation would be take longer because it has
to do redundancy checks. And I still feel there should be a way to
create a UNIQUE INDEX without waiting for the redundancy check.
Do the inserts in a temp table memory based inside a transaction,
create the index on this table and then pass the table to file. Also,
you may check/tweak SQLite, as page size (put 32768 byte, default is
1024), increase page cache (default is 2000, each page has 1.5*page
bytes size, so do maths how many you can have), temp store (use
memory). If not enough speed, you must or duplicate hardware (2
SQLite works faster than one) or think about change to another
database, i suggest using DB2, Valentina (a fast-low price database)
or any other commercial database.
For your mail i think you have everything planned, but just for
curiosity, how do you plan to pass data from hardware (800 MB/s +
Database data) to a raid?? A GigEthernet has 1000 Mb/s (so, 128
MB/s), a PCI-X based solution has 1.3 to 2.6 Gb/s (so 150-300 MB/s),
and similar for SATA and PCIe(x16). Even a direct Hypertransport to
the raid has 3.2 Gb/s (. Note that i'm not asking for the raid, just
for the SQLite hardware to Raid connection.
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
