Serena Lien wrote:
Thanks for the response.
-- 1 Gbyte is not at all too big for Sqlite to handle in one file.
True, but I think too big for an in-memory database - I was considering
copying all my source data from separate databases into one table in an
in-memory database so that I could access it easier later. I don't really
know what the size I need to copy would be, it would depend on some sort of
search to work out what input files to use.
Eg Say each input database corresponds to some experimental data associated
with one subject. I may have thousands of these. In the application I am
writing, I may want to do a query to find all subjects with some sort of
common condition - so this may return a lot of database files, and/or
have a
lot of data (size on disk) associated with it.
-- Some portion (large??) of the performance advantages of transactions
will
be lost in opening/closing files, even if transactions were allowed
across
multiple files.
The transactions are hopefully for optimizing the insertion of data into my
results database file, not for reading data from my multiple source
files, I
don't need to write to the sources.
For these reasons, might you consider using a single database rather than
multiple files if you're up against performance bottlenecks?
Perhaps - I thought if I needed to work with lots and lots of data in the
future, rather than having everything in one database, it would be
easier to
manage in separate files, because I wouldn't always need to access all the
experimental data all the time.
Serena.
Why7 copy to an im memory database? Why not just create a regular
database as your output. The is no reason you cannot have more than one
DB connection current in any process or thread.
How about -
Attach your input databases to make one large DB
select your output data
insert output data into the output DB
Of course you want to use transactions, and to limit the transaction
size to a reasonable number.
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
