Hi Antonio,
You are right, I don´t need to load the entire table into memory.
The fourth column has multidimensional cells and when I read a single row
from every cell in the column, I almost fill the workstation memory.
I didn´t expect that process to use so much memory, but the fact is that it
uses it.
May be I didn´t explain very well last time.
Thank you,
Juanma
2012/8/5 Antonio Valentino <antonio.valent...@tiscali.it>
> Hi Juan Manuel,
>
> Il 04/08/2012 01:55, Juan Manuel Vázquez Tovar ha scritto:
> > Hello all,
> >
> > I´m managing a file close to 26 Gb size. It´s main structure is a table
> > with a bit more than 8 million rows. The table is made by four columns,
> the
> > first two columns store names, the 3rd one has a 53 items array in each
> > cell and the last column has a 133x6 matrix in each cell.
> > I use to work with a Linux workstation with 24 Gb. My usual way of
> working
> > with the file is to retrieve, from each cell in the 4th column of the
> > table, the same row from the 133x6 matrix.
> > I store the information in a bumpy array with shape 8e6x6. In this
> process
> > I almost use the whole workstation memory.
> > Is there anyway to optimize the memory usage?
>
> I'm not sure to understand.
> My impression is that you do not actually need to have the entire 8e6x6
> matrix in memory at once, is it correct?
>
> In that case you could simply try to load less data using something like
>
> data = table.read(0, 5e7, field='name of the 4-th field')
> process(data)
> data = table.read(5e7, 1e8, field='name of the 4-th field')
> process(data)
>
> See also [1] and [2].
>
> Does it make sense for you?
>
>
> [1]
> http://pytables.github.com/usersguide/libref.html#table-methods-reading
> [2] http://pytables.github.com/usersguide/libref.html#tables.Table.read
>
> > If not, I have been thinking about splitting the file.
> >
> > Thank you,
> >
> > Juanma
>
>
> cheers
>
> --
> Antonio Valentino
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
