On 5/22/06, Micha Bieber <[EMAIL PROTECTED]> wrote:
> Interesting, but it doesn't have any adhoc query capability:
Take in consideration, that this is not the whole truth - simply
selecting and getting what the HDF5 authors call a 'hyperslab' is a
great feature and an example for a simple query
> Interesting, but it doesn't have any adhoc query capability:
Take in consideration, that this is not the whole truth - simply
selecting and getting what the HDF5 authors call a 'hyperslab' is a
great feature and an example for a simple query task. This special task
alone might become an essentia
Thank you for answering the question! Like I said in my other post, the
index creation is about 20% faster than the unique index creation in my
latest tests of an index of two INTEGERs. 20% out of several hours is
significant.
So index names are unique for the whole database? I was thinking they j
> I am simply curious. This sounds like an amazing
> engineering challenge. If it is not a secret, can you
> describe what this data represents and how it will be used?
Genomics. Look up "Smith-Waterman" or "Needleman-Wunsch-Sellers" on the web.
> What is the ultimate source of this d
On 5/22/06, Brannon King <[EMAIL PROTECTED]> wrote:
> _Personally_ I think, this sounds like a task not quite
> fitting in sqlites (probably any 'standard' databases) realm.
> This is a bit off-topic in this group, but because you
> mention sub-boxes - did you ever look into more specialized
> fi
> _Personally_ I think, this sounds like a task not quite
> fitting in sqlites (probably any 'standard' databases) realm.
> This is a bit off-topic in this group, but because you
> mention sub-boxes - did you ever look into more specialized
> file-formats like HDF5:
>
> http://hdf.ncsa.uiuc.ed
On 5/22/06, Dennis Jenkins <[EMAIL PROTECTED]> wrote:
Jay Sprenkle wrote:
>
> Me too!
>
> The two largest database setups I've worked with:
> the total of all the call records for a major phone company,
> and the cumulative records of most of the drugs bought in
> the United States, don't add up
Monday, May 22, 2006, 15:17:21, Jay Sprenkle wrote:
> On 5/22/06, Dennis Jenkins <[EMAIL PROTECTED]> wrote:
>> Brannon King wrote:
>> > The benefits I'm trying to get out of sqlite are the data queries. I
>> > collect a large, sparse 2D array from hardware. The hardware device is
>> > giving me a
Jay Sprenkle wrote:
>
> Me too!
>
> The two largest database setups I've worked with:
> the total of all the call records for a major phone company,
> and the cumulative records of most of the drugs bought in
> the United States, don't add up to as much as this.
> What are you sampling at 1/200th o
On 5/22/06, Dennis Jenkins <[EMAIL PROTECTED]> wrote:
Brannon King wrote:
> The benefits I'm trying to get out of sqlite are the data queries. I
> collect a large, sparse 2D array from hardware. The hardware device is
> giving me a few GB of data data at 200MB/s. Future hardware versions
> will b
Brannon King wrote:
> The benefits I'm trying to get out of sqlite are the data queries. I
> collect a large, sparse 2D array from hardware. The hardware device is
> giving me a few GB of data data at 200MB/s. Future hardware versions
> will be four times that fast and give me terabytes of data. Af
"Brannon King" <[EMAIL PROTECTED]> wrote:
> The statement
> CREATE UNIQUE INDEX primkey ON 4dData (x,y,z)
> is too slow for me.
>
> Is there some way I can create that unique index without the overhead of
> uniqueness checking? I know all my triples are unique when used =
> together: I
> generated
On 5/20/06, Brannon King <[EMAIL PROTECTED]> wrote:
The benefits I'm trying to get out of sqlite are the data queries. I
collect a large, sparse 2D array from hardware. The hardware device is
giving me a few GB of data data at 200MB/s. Future hardware versions
will be four times that fast and giv
Is that 800MB/s sustained? Is it constant, 24/7? Or is it bursty data?
If it's bursty, you could buffer the inserts in an append only log.
Insertion to the SQL(ite) database can be done asynchronously. Writing
to unstructured, append only log files can be done at pretty much disk
I/O speed.
On Sat, 20 May 2006, Brannon King wrote:
John Stanton wrote:
You don't seem to need a data manipulation system like Sqlite, more a form
of high volume storage. Do you really need elaborate SQL, journalling,
ROLLBACK and assured disk storage?
Di you consider some form of hashed storage, perh
On Sat, 20 May 2006, Brannon King wrote:
[...] I don't need journaling or rollback. I'd love a way to shut them
off. [...]
In that case zou can at least do a PRAGMA synchronous = OFF; to improve
speed at the cost of safety. See http://www.sqlite.org/pragma.html#modify
for more information.
Brannon King wrote:
John Stanton wrote:
You don't seem to need a data manipulation system like Sqlite, more a
form of high volume storage. Do you really need elaborate SQL,
journalling, ROLLBACK and assured disk storage?
Di you consider some form of hashed storage, perhaps linear hashing,
John Stanton wrote:
You don't seem to need a data manipulation system like Sqlite, more a
form of high volume storage. Do you really need elaborate SQL,
journalling, ROLLBACK and assured disk storage?
Di you consider some form of hashed storage, perhaps linear hashing,
to build a compact and
You don't seem to need a data manipulation system like Sqlite, more a
form of high volume storage. Do you really need elaborate SQL,
journalling, ROLLBACK and assured disk storage?
Di you consider some form of hashed storage, perhaps linear hashing, to
build a compact and high performance ass
For your mail i think you have everything planned, but just for
curiosity, how do you plan to pass data from hardware (800 MB/s +
Database data) to a raid?? A GigEthernet has 1000 Mb/s (so, 128 MB/s),
a PCI-X based solution has 1.3 to 2.6 Gb/s (so 150-300 MB/s), and
similar for SATA and PCIe(
At 21:28 20/05/2006, you wrote:
The benefits I'm trying to get out of sqlite are the data queries. I
collect a large, sparse 2D array from hardware. The hardware device
is giving me a few GB of data data at 200MB/s. Future hardware
versions will be four times that fast and give me terabytes of
The benefits I'm trying to get out of sqlite are the data queries. I
collect a large, sparse 2D array from hardware. The hardware device is
giving me a few GB of data data at 200MB/s. Future hardware versions
will be four times that fast and give me terabytes of data. After I have
the data, I t
On 5/18/06, Brannon King <[EMAIL PROTECTED]> wrote:
The statement
CREATE UNIQUE INDEX primkey ON 4dData (x,y,z)
is too slow for me. It takes 13 sec on 700k records on my Athlon XP 2GHz.
That's about four seconds longer than it took me to put the data in there.
The two times together are still fa
23 matches
Mail list logo
