On 2008-08-31 15:15, mark wrote: > Hi there, > > I need to extract data from text files (~4 GB) on this data some > operations are performed like avg, max, min, group etc. The result is > formated and written in some other text files (some KB). > > I currently think about database tools might be suitable for this. I > would just write the import from the text files and ... the tool does > the rest. The only problem I can imagine is that this would not be > fast enough. But I would give it a shoot. > Unfortunately I have only some knowledge of SQLite which is not an > option here. > > Some additional requirements I can think of are: > - Python (I want to hone my programming skills too) > - Python-only (no C-lib) for simplicity (installation, portability). > Therefore SQLite is not an option > - must be fast > - I like SQL (select a, b from ...) this would be nice (row[..] + ... > is a little hard getting used to) > > So far I found PyDBLite, PyTables, Buzhug but they are difficult to > compare for a beginner.
You could use Gadfly for this since it is pure Python and provides a standard Python DB-API interface: http://gadfly.sourceforge.net/ (the C extensions are optional to speedup processing) This is the SQL subset it supports: http://gadfly.sourceforge.net/sql.html Another option is SnakeSQL: http://pythonweb.org/projects/snakesql/ but I've never used that one, so can't judge its quality. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 01 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 -- http://mail.python.org/mailman/listinfo/python-list