Hi all, I found numpy.memmap to be very suitable when matrices larger than the physical memory are required.
1. included in standard numpy installation 2. very low learning curve. pyTables seems to be more suitable but , i somehow found the learning curve too steep . Also pyTables needs lot of initializations before anything can be done with it as compared to memmap. The above reason made me use memmap over pyTable. On Sat, Jan 15, 2011 at 11:34 PM, kunal ghosh <kunal...@gmail.com> wrote: > Thanks Santosh , > > This stack overflow thread indeed discusses the exact same problem i have. > Wonder how i missed it :) in my preliminary searches. > > thanks again ! > > > On Sat, Jan 15, 2011 at 11:12 PM, Santosh Rajan <santra...@gmail.com>wrote: > >> Hope this helps >> >> http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices >> >> On Sat, Jan 15, 2011 at 10:11 PM, kunal ghosh <kunal...@gmail.com> wrote: >> > Hi all, >> > while implementing Locality Preserving Projections , >> > at one point i have to perform X L X.transpose() >> > these matrices are large (32256 x 32256) so i get "out of memory" error. >> > >> > I assume, as the dataset gets larger one would come across this problem >> , >> > how would >> > one go about solving this ? Is there a common trick that is used to deal >> > with such problems ? >> > Or the workstation calculating these problems needs to have HUGE >> amounts of >> > physical memory ? >> > >> > I am using python and numpy / scipy >> > >> > -- >> > regards >> > ------- >> > Kunal Ghosh >> > Dept of Computer Sc. & Engineering. >> > Sir MVIT >> > Bangalore,India >> > >> > permalink: member.acm.org/~kunal.t2 >> > Blog:kunalghosh.wordpress.com >> > Website:www.kunalghosh.net46.net >> > _______________________________________________ >> > BangPypers mailing list >> > BangPypers@python.org >> > http://mail.python.org/mailman/listinfo/bangpypers >> > >> >> >> >> -- >> http://about.me/santosh.rajan >> _______________________________________________ >> BangPypers mailing list >> BangPypers@python.org >> http://mail.python.org/mailman/listinfo/bangpypers >> > > > > -- > regards > ------- > Kunal Ghosh > Dept of Computer Sc. & Engineering. > Sir MVIT > Bangalore,India > > permalink: member.acm.org/~kunal.t2 > Blog:kunalghosh.wordpress.com > Website:www.kunalghosh.net46.net > > > -- regards ------- Kunal Ghosh Dept of Computer Sc. & Engineering. Sir MVIT Bangalore,India permalink: member.acm.org/~kunal.t2 Blog:kunalghosh.wordpress.com Website:www.kunalghosh.net46.net _______________________________________________ BangPypers mailing list BangPypers@python.org http://mail.python.org/mailman/listinfo/bangpypers
