Excerpts from Nathaniel Smith's message of Mon Feb 27 12:07:11 -0500 2012: > On Mon, Feb 27, 2012 at 2:44 PM, Erin Sheldon <erin.shel...@gmail.com> wrote: > > What I've got is a solution for writing and reading structured arrays to > > and from files, both in text files and binary files. It is written in C > > and python. It allows reading arbitrary subsets of the data efficiently > > without reading in the whole file. It defines a class Recfile that > > exposes an array like interface for reading, e.g. x=rf[columns][rows]. > > What format do you use for binary data? Something tiled? I don't > understand how you can read in a single column of a standard text or > mmap-style binary file any more efficiently than by reading the whole > file.
For binary, I just seek to the appropriate bytes on disk and read them, no mmap. The user must have input an accurate dtype describing rows in the file of course. This saves a lot of memory and time on big files if you just need small subsets. For ascii, the approach is similar except care must be taken when skipping over unread fields and rows. For writing binary, I just tofile() so the bytes correspond directly between array and file. For ascii, I use the appropriate formats for each type. Does this answer your question? -e -- Erin Scott Sheldon Brookhaven National Laboratory _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
