Re: [Tutor] Optimal solution in dealing with huge databases in python

Shadab Sayani [EMAIL PROTECTED] wrote The data I need to deal with is in 100s of GB. I am using postgresql backend and SQLALCHEMY ORM. All ORMs will introduce a significant performance hit. If you really need high speed, and populating a 100G+ database probably is included in that, then you

Re: [Tutor] Optimal solution in dealing with huge databases in python

Hi Alan, Thank you very much for immediate response.I didnt get the point of loading the data using SQL.What does that mean?Does it mean Bulk file load.If so then there will I/O performance hit right.Writing the parsed tokens in the csv format in a file.After that only I can bulk load this file.

Re: [Tutor] Optimal solution in dealing with huge databases in python

This page has some info on C/C++ for PostgreSQL: http://www.postgresql.org/docs/8.0/interactive/xfunc-c.html Shadab Sayani wrote: Do have any idea about the C api for Postgresql and some documentation to use it? -- == Chuck

Re: [Tutor] Optimal solution in dealing with huge databases in python

On Wednesday 24 January 2007 19:22, Shadab Sayani wrote: Hi, I am working in a biodatabases project.The data I need to deal with is in 100s of GB.I am using postgresql backend and SQLALCHEMY ORM.I need to read the bio datafiles and parse them and then store them in database.I am in the

Re: [Tutor] Optimal solution in dealing with huge databases in python

Alan Gauld wrote: Shadab Sayani [EMAIL PROTECTED] wrote The data I need to deal with is in 100s of GB. I am using postgresql backend and SQLALCHEMY ORM. All ORMs will introduce a significant performance hit. If you really need high speed, and populating a 100G+ database probably is

Re: [Tutor] Optimal solution in dealing with huge databases in python

Hi, I got your point.But before inserting data I need to store it into a file in a format supported by postgresql.Wont this operation incur a performance hit as it includes writing to a file which is on disk? Thanks and Regards, Shadab. johnf [EMAIL PROTECTED] wrote: On Wednesday

Re: [Tutor] Optimal solution in dealing with huge databases in python

You could try Pytables: http://www.pytables.org/moin It is not a database, but a storage library for large amounts of data. Regards, Eike. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor

[Tutor] Optimal solution in dealing with huge databases in python

Hi, I am working in a biodatabases project.The data I need to deal with is in 100s of GB.I am using postgresql backend and SQLALCHEMY ORM.I need to read the bio datafiles and parse them and then store them in database.I am in the process of storing them. I used the session,flush concept