Do not put tick by tick data in a database, The stuff is not relational, it
is time series, FAME was one on the few that could deal it but I think it
died. A database does not help research or organizing the data, it gets in
the way. you would be better off timestamping the data to the millisecond
and making a simple structure for trade and quote records that can run them
together in time order and making a flat ACSII file for each symbol per day,
depending on the access pattern use zlib to compress the file(s) after a few
days(should get 8:1 compression and fast read access). You need a few custom
filters if you do not want to pull back the whole symbol data for the day or
to handle splits. Now using A database to mange the files and provide
summary(open,close,high,low,etc...) information for screening is fine but
not the raw data.
MJM
----- Original Message -----
From: "Tom Wheeler" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, July 19, 2001 4:15 PM
Subject: Questions about extremely large database support
> We are beginning a project and need to choose a database. We have been
> using MS Visual FoxPro, which is like mySQL in that it is very fast and
> ISAM-based, but is used more often as a file database rather than
> client/server. Unfortunately, we are quickly approaching FoxPro's 2GB
table
> size limit, so we cannot continue using it.
>
> Our two most important requirements for the database engine are speed and
> scalability. We will be inserting probably 5,000,000 records per day into
> our database, and will maintain around 2TB of data. Of course, these
> figures will grow in time (probably about 40% per year). We will be
> accessing the data using custom C++ components, and at this stage, our
data
> model/design is fairly flexible. We would like to be able to scale
easily,
> potentially by splitting databases up across multiple machines.
> My questions are:
> Is mySQL up to such a task? I have been using mySQL for four years
> now, but have never used it in a project of this magnitude. We'll be
> handling financial data in the database, so integrity is important.
>
> Where can I find information about very large databases with mySQL?
>
> Is there a really efficient way to implement a function like
> indexseek() in FoxPro? This function will simply check an index to tell
you
> if a record with that key exists. Sort of like "select count(id) from Foo
> where id=1" except that it doesn't actually fetch the field value and it
> just checks the index file, not the data file.
>
> Thanks for your help. If you would prefer to take this thread offline
from
> the list, you may e-mail me directly at [EMAIL PROTECTED]
>
> tw
>
> ---------------------------------------------
> Tom Wheeler
> Software Engineer
> Teralogix, Inc.
> [EMAIL PROTECTED]
>
> ---------------------------------------------------------------------
> Before posting, please check:
> http://www.mysql.com/manual.php (the manual)
> http://lists.mysql.com/ (the list archive)
>
> To request this thread, e-mail <[EMAIL PROTECTED]>
> To unsubscribe, e-mail
<[EMAIL PROTECTED]>
> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
>
>
---------------------------------------------------------------------
Before posting, please check:
http://www.mysql.com/manual.php (the manual)
http://lists.mysql.com/ (the list archive)
To request this thread, e-mail <[EMAIL PROTECTED]>
To unsubscribe, e-mail <[EMAIL PROTECTED]>
Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
