Re: Transforming ascii file (pseduo database) into proper database

In article <[EMAIL PROTECTED]>, p. <[EMAIL PROTECTED]> wrote: >I need to take a series of ascii files and transform the data >contained therein so that it can be inserted into an existing >database. The ascii files are just a series of lines, each line >containing fields separated by '|' character.

Re: Transforming ascii file (pseduo database) into proper database

Thanks to all for the ideas. I am familiar with external sorting. Hadn't considered it though. Will definitely be giving that a go, and then merging. Again, thanks all. -- http://mail.python.org/mailman/listinfo/python-list

Re: Transforming ascii file (pseduo database) into proper database

p. a écrit : > I need to take a series of ascii files and transform the data > contained therein so that it can be inserted into an existing > database. The ascii files are just a series of lines, each line > containing fields separated by '|' character. Relations amongst the > data in the various

Re: Transforming ascii file (pseduo database) into proper database

"p." <[EMAIL PROTECTED]> writes: > So as an exercise, lets assume 800MB file, each line of data taking up > roughly 150B (guesstimate - based on examination of sample data)...so > roughly 5.3 million unique IDs. I still don't understand what the problem is. Are you familiar with the concept of ex

Re: Transforming ascii file (pseduo database) into proper database

So in answer to some of the questions: - There are about 15 files, each roughly representing a table. - Within the files, each line represents a record. - The formatting for the lines is like so: File1: somval1|ID|someval2|someval3|etc. File2: ID|someval1|someval2|somewal3|etc. Where ID is the o

Re: Transforming ascii file (pseduo database) into proper database

On Jan 21, 4:45 pm, Paul Rubin wrote: > "p." <[EMAIL PROTECTED]> writes: > > 1. Has anyone done anything like this before, and if so, do you have > > any advice? > > Sort all the files with an external sort utility (e.g. unix sort), so > that records with the same key are

Re: Transforming ascii file (pseduo database) into proper database

"p." <[EMAIL PROTECTED]> writes: > 1. Has anyone done anything like this before, and if so, do you have > any advice? Sort all the files with an external sort utility (e.g. unix sort), so that records with the same key are all brought together. Then you can process the files sequentially. -- htt

Re: Transforming ascii file (pseduo database) into proper database

On Jan 22, 7:51 am, "p." <[EMAIL PROTECTED]> wrote: > I need to take a series of ascii files and transform the data > contained therein so that it can be inserted into an existing > database. The ascii files are just a series of lines, each line > containing fields separated by '|' character. Relat

Re: Transforming ascii file (pseduo database) into proper database

> I need to take a series of ascii files and transform the data > contained therein so that it can be inserted into an existing > database. [snip] > I need to transform the data from the files before inserting > into the database. Now, this would all be relatively simple if > not for the followin

Transforming ascii file (pseduo database) into proper database

I need to take a series of ascii files and transform the data contained therein so that it can be inserted into an existing database. The ascii files are just a series of lines, each line containing fields separated by '|' character. Relations amongst the data in the various files are denoted throu