Hi Vineet,
2010/8/13 Vineet Jain <[email protected]>
> I have the following scrip which transforms several pytables files by
> changing the format of one field in a table from a StringCol to Int32
> col. I have a couple of problems:
>
> 1. The memory usage of the process keeps growing and then dies. I have
> to restart it several times
> 2. Is there any way to do this without converting it to a python list
> and make it faster.
>
After having a quick look at your code, I think the problem is precisely
what you are suggesting: you need to get rid of the list so as to not bloat
your memory unnecessarily. Keep reading for a proposed solution.
#save symbol mapping
> rowsToBeInserted = []
> for row in mapping.tblData.iterrows():
> row[2] = int(row[2])
> rowsToBeInserted.append((row[0], row[1], row[2], row[3]))
> if rowsToBeInserted:
> mapping2.tblData.append(rowsToBeInserted)
> mapping2.tblData.flush()
>
I'd suggest replacing the code above by something like:
#save symbol mapping
dsttable = mapping2.tblData
for row in mapping.tblData.iterrows():
r = list(row) # convert row into a list
r[2] = int(r[2])
r = tuple(r) # convert again into a tuple
dsttable.append([r]) # append converted row to dsttable
dsttable.flush()
I have not checked the code, but I hope you get the idea.
--
Francesc Alted
------------------------------------------------------------------------------
This SF.net email is sponsored by
Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Pytables-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pytables-users
