you should have a read here/ http://wesmckinney.com/blog/?p=543
going below the 2x memory usage on read in is non trivial and costly in terms of performance > On Oct 26, 2014, at 4:46 AM, Saullo Castro <saullogiov...@gmail.com> wrote: > > I would like to start working on a memory efficient alternative for > np.loadtxt and np.genfromtxt that uses arrays instead of lists to store the > data while the file iterator is exhausted. > > The motivation came from this SO question: > > http://stackoverflow.com/q/26569852/832621 > > where for huge arrays the current NumPy ASCII readers are really slow and > require ~6 times more memory. This case I tested with Pandas' read_csv() and > it required 2 times more memory. > > I would be glad if you could share your experience on this matter. > > Greetings, > Saullo > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
