On Wed, Jul 1, 2009 at 7:48 AM, Barak, Ron<ron.ba...@lsi.com> wrote: > Hi, > > I think I'm up against a limitation in cStringIO.StringIO(), but could not > find any clues on the web, especially not in > http://docs.python.org/library/stringio.html. > > What I have is the following function: > > def concatenate_files(self,filename_array): > import types > > f = cStringIO.StringIO() > for filename in filename_array: > log_stream = LogStream(filename) > if not isinstance(log_stream, types.NoneType): > try: > string_ = log_stream.input_file.read() > except AttributeError, e: > sys.stderr.write("AttributeError: "+str(e)+"\n") > sys.stderr.write("log_stream: > "+str(log_stream)+"\n") > else: > return(None) > > f.write(string_) > return (f) > > And, when the list of files - in filename_array - is pointing to long/large > enough files, I get the exception: > > Traceback (most recent call last): > File "svm_ts_tool_in_progress.py", line 244, in OnSelectCell > self.InspectorViewController(row,col) > File "svm_ts_tool_in_progress.py", line 1216, in InspectorViewController > self.InspectorePaneGetRecords(self.req_table, rec) > File "svm_ts_tool_in_progress.py", line 1315, in InspectorePaneGetRecords > log_stream = > ConcatenatedLogStream(self.events_dict[req_table]["db_dict"].keys()) > File "c:\Documents and > Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 31, in > __init__ > self.input_file = self.concatenate_files(filename_array) > File "c:\Documents and > Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 47, in > concatenate_files > string_ = log_stream.input_file.read() > MemoryError > > Anyone knows whet's the limitation on cStringIO.StringIO() objects ? > Could you suggest a better way to create a string that contains the > concatenation of several big files ? > > Thanks, > Ron. > > -- > http://mail.python.org/mailman/listinfo/python-list > >
MemoryError means you're out of memory. Try using the operating system commands for file concatenation, i.e. "cat" on *nix, dunno how on windows. Or just do whatever it is you're doing in a different way that doesn't require having all the data in memory at once. Also, this: not isinstance(log_stream, types.NoneType) should just be this: log_stream is not None and return is not a function, leave out the ()'s around the expression you're returning, and if you're returning None, you can leave out the None as it is the default return value anyway. Generally speaking don't do module imports in functions. -- http://mail.python.org/mailman/listinfo/python-list