Thank you for responding Amaury.
I looked at the code for memmap.py. The core of it comes down to these two
lines of code:
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,
offset=offset, order=order)
It seems to me one needs to modify ndarray.__new__ so it can take a named
buffer= argument, where that argument can be an mmap object. Currently
the python ndarray.__new__ can accept such an argument, but the numpypy one
does not appear to. When I try to figure out the changes to
interp_numarray.py to make it work, it goes way over my head.
Thoughts?
Mike
On Fri, Jun 14, 2013 at 11:08 AM, Amaury Forgeot d'Arc
<[email protected]>wrote:
> Hi, this is definitely doable, but needs some work.
> Here are the tasks I identified:
>
> - copy "memmap.py" from the official numpy, and write a unit test (there
> is a nice docstring in the previous file)
>
> - add support in numpy for buffers with a fixed address
> (in interp-level terms: a RWBuffer with a get_raw_address() method)
>
> - have buffer(mmap) return such a buffer, very very similar to
> array.ArrayType.
>
> The three tasks are quite independent, not too difficult, and could be a
> nice start for newcomers...
> I'll be happy to help.
>
>
>
> 2013/6/14 Mike Beller <[email protected]>
>
>> Hello
>>
>> Do you think it is likely that the memap capabilities of numpy will find
>> their way in to numpypy any time soon?
>>
>> It seems to me that some people think memap is a relatively unimportant
>> aspect of numpy. But I do not think so. Because of the way the linux IO
>> subsystem and virtual memory systems interact, memap enables numpy to have
>> high performance access to very large data sets -- it helps make numpy
>> relevant to "Big Data".
>>
>> The code to allow numpy to support memap doesn't seem very large. But,
>> while I have tried reading through the code, I really can not tell whether
>> the same is true for numpypy, or whether it is a large endeavor (for
>> example, due to some kind of pypy memory management architectural issue).
>>
>> I'm interested in any input on this.
>>
>> Mike Beller
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> [email protected]
>> http://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
>
> --
> Amaury Forgeot d'Arc
>
_______________________________________________
pypy-dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/pypy-dev
