Ken wrote:
I do have the option of either putting the data into sqlite at the start (when its read) or putting it into a flat file and then Later loading it into a sqlite db via a downstream job.
A great deal of the data columns are simple numeric values and thats where sqlite really shines in that is portable between systems having differing endianness.
Here is a summary of the entire processing where a1/b1 are different
processes and probably differnt host platforms.
a1, read from producing system, generate output data (be it flat file or sqlite).
a2. Compress data file.
a3. Transfer compressed datafile to target system.
a4. Goto a1
b1. Receive datafile and uncompress.
b2. Read datafile and load into Master DB. (This might just be a simple
attach).
b3. Massage data in Master db ???
b4. Read and process data from MasterDb. Delete or mark as deleteable.
b5. Delete processed data from MasterDb. (this could be in a seperate thread).
b6. Goto step b1.
The nice thing about simply attaching as a DB in step b2 is that when all data is processed from step b4 then step b5 to purge is a simple detach and operating system unlink for the underlying datafile. Which I suspect will be infinately faster than a sql delete command.
Ken,
If you are always going to move the data into an sqlite database at some
point, then I would think you would want to do that on system a when the
data is read. Unless system a is so much slower than system b that the
extra work is a burden on system a, whereas it is insignificant on
system b.
In either case your timing of the exchange file write function in system
a only is incomplete. You are neglecting the time it takes system b to
open and read the exchange file. If you sum those two times then you can
see which solution provides the fastest overall delivery of the data to
system b in the required format.
Dennis Cote
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
