On Dec 12, 2015 10:53 AM, "Mathieu Dubois" <mathieu.dub...@icm-institute.org> wrote: > > Le 11/12/2015 11:22, Sturla Molden a écrit : >> >> Mathieu Dubois <mathieu.dub...@icm-institute.org> wrote: >> >>> The point is precisely that, you can't do memory mapping with Npz files >>> (while it works with Npy files). >> >> The operating system can memory map any file. But as npz-files are >> compressed, you will need to uncompress the contents in your memory mapping >> to make sense of it. > > We agree on that. The goal is to be able to create a np.memmap array from an Npz file. > > >> I would suggest you use PyTables instead of npz-files. >> It allows on the fly compression and uncompression (via blosc) and will >> probably do what you want. > > Yes I know I can use other solutions. The point is that np.load silently ignore the mmap option so I wanted to discuss ways to improve this.
I can see a good argument for transitioning to a rule where mmap=False doesn't mmap, mmap=True mmaps if the file is uncompressed and raises an error for compressed files, and mmap="if-possible" gives the current behavior. (It's even possible that the current code would already accept "if-possible" as a alias for True, which would make the transition easier.) Or maybe "never"/"always"/"if-possible" would be better for type consistency reasons, while deprecating the use of bools altogether. But this transition might be a bit more of a hassle, since these definitely won't work on older numpy's. Silently creating a massive temporary file doesn't seem like a great idea to me in any case. Creating a temporary file + mmaping it is essentially equivalent to just loading the data into swappable RAM, except that the swap case is guaranteed not to accidentally leave a massive temp file lying around afterwards. -n
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
