On 08/07/2017 05:01 PM, Nisoli Isaia wrote: > Dear all, > I have a question about the behaviour of > > y = np.array(x, copy=False, dtype='float32') > > when x is a memmap. If we check the memmap attribute of mmap > > print "mmap attribute", y._mmap > > numpy tells us that y is not a memmap. > But the following code snippet crashes the python interpreter > > # opens the memmap > with open(filename,'r+b') as f: > mm = mmap.mmap(f.fileno(),0) > x = np.frombuffer(mm, dtype='float32') > > # builds an array from the memmap, with the option copy=False > y = np.array(x, copy=False, dtype='float32') > print "before", y > > # closes the file > mm.close() > print "after", y > > In my code I use memmaps to share read-only objects when doing parallel > processing > and the behaviour of np.array, even if not consistent, it's desirable. > I share scipy sparse matrices over many processes and if np.array would > make a copy > when dealing with memmaps this would force me to rewrite part of the sparse > matrices > code. > Would it be possible in the future releases of numpy to have np.array > check, > if copy is false, if y is a memmap and in that case return a full memmap > object > instead of slicing it?
This does appear to be a bug in numpy or mmap. Probably the solution isn't to make mmaps a special case, rather we should fix a bug somewhere in the use of the PEP3118 interface. I've opened an issue on github for your issue: https://github.com/numpy/numpy/issues/9537 It seems to me that the "correct" behavior may be for it to me impossible to close the memmap while pointers to it exist; this is the behavior for `memoryview`s of mmaps. That is, your line `mm.close()` shoud raise an error `BufferError: cannot close exported pointers exist`. > Best wishes > Isaia > > P.S. A longer account of the issue may be found on my university blog > http://www.im.ufrj.br/nisoli/blog/?p=131 > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
