Re: [Python-Dev] ctypes, memory mapped files and context manager

[eryk sun]

On Sun, Jan 8, 2017 at 8:25 AM, Armin Rigo  wrote:
>
> c_raw = ctypes.PYFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)(lambda p: p)

Use ctypes.addressof.

> addr = c_raw(ctypes.pointer(T.from_buffer(m)))
> b = ctypes.cast(addr, ctypes.POINTER(T)).contents

ctypes.cast uses an FFI call. In this case you can more simply use from_address:

b = T.from_address(ctypes.addressof(T.from_buffer(m)))

There's no supporting connection between b and m. If m was allocated
from a heap/pool/freelist, as opposed to a separate mmap
(VirtualAlloc) call, then you won't necessarily get a segfault (access
violation) if b is used after m has been deallocated or internally
realloc'd. It can lead to corrupt data and difficult to diagnose
errors. You're lucky if it segfaults.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Armin Rigo]

Hi Hans-Peter,

On 6 January 2017 at 00:28, Hans-Peter Jansen  wrote:
> Leaves the question, how stable this "interface" is?

Another way to jump through hoops:

c_raw = ctypes.PYFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)(lambda p: p)

addr = c_raw(ctypes.pointer(T.from_buffer(m)))
b = ctypes.cast(addr, ctypes.POINTER(T)).contents

These lines give an object 'b' that is equivalent to
'T.from_buffer(m)', but doesn't hold any reference or any "opened
buffer" state to the original 'm'.  Your context manager can yield
that.  It should prevent all BufferErrors, at the price of segfaulting
if used incorrectly.  This means in your case that ``with
map_struct(..) as a:`` should not continue to use ``a`` after the
``with`` statement, which is pretty natural anyway.

(The same issue occurs with cffi instead of ctypes, but in this case a
simple cast is enough to detach the memoryview, instead of the hack
above.)


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[eryk sun]

On Thu, Jan 5, 2017 at 11:28 PM, Hans-Peter Jansen  wrote:
> Leaves the question, how stable this "interface" is?
> Accessing _objects here belongs to voodoo programming practices of course, but
> the magic is locally limited to just two lines of code, which is acceptable in
> order to get this context manager working without messing with the rest of the
> code.

My intent was not to suggest that anyone directly use the _objects
value / dict  in production code. It's a private implementation
detail. I was demonstrating the problem of simply releasing the buffer
and the large number of checks that would be required if b_ptr is
cleared. It would be simpler for a release() method to allocate new
memory for the object and set the b_needsfree flag, but this may hide
bugs. Operating on a released object should raise an exception.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

On Freitag, 6. Januar 2017 00:28:37 Hans-Peter Jansen wrote:
> Hi Eryk,
> 
> This is exactly, what I was after:
> 
> @contextmanager
> def cstructmap(cstruct, mm, offset = 0):
> # resize the mmap (and backing file), if structure exceeds mmap size
> # mmap size must be aligned to mmap.PAGESIZE
> cssize = ctypes.sizeof(cstruct)
> if offset + cssize > mm.size():
> newsize = align(offset + cssize, mmap.PAGESIZE)
> mm.resize(newsize)
> cmap = cstruct.from_buffer(mm, offset)
> try:
> yield cmap
> finally:
> for mv in cmap._objects.values():
if isinstance(mv, memoryview):
mv.release()

It happens, that _objects contain other objects as well...

Cheers,
Pete
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Hi Eryk,

On Donnerstag, 5. Januar 2017 15:30:33 eryk sun wrote:
> 
> > manager introduce a layer of indirection:
> I think that's the best you can do with the current state of ctypes.
> 
> from_buffer was made safer in Python 3 by ensuring it keeps a
> memoryview reference in the _objects attribute (i.e.
> CDataObject.b_objects). Hans-Peter's problem is a consequence of this
> reference. Simply calling release() on the underlying memoryview is
> 
> unsafe. For example:
> >>> b = bytearray(2**20)
> >>> a = ctypes.c_char.from_buffer(b)
> >>> a._objects
> 
> 
> 
> >>> a._objects.release()
> >>> del b
> >>> a.value
> 
> Segmentation fault (core dumped)

This is exactly, what I was after:

@contextmanager
def cstructmap(cstruct, mm, offset = 0):
# resize the mmap (and backing file), if structure exceeds mmap size
# mmap size must be aligned to mmap.PAGESIZE
cssize = ctypes.sizeof(cstruct)
if offset + cssize > mm.size():
newsize = align(offset + cssize, mmap.PAGESIZE)
mm.resize(newsize)
cmap = cstruct.from_buffer(mm, offset)
try:
yield cmap
finally:
for mv in cmap._objects.values():
mv.release()

See also:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx3-py

While technically possible (which is a surprise for me on its own), nothing 
should access the with variable after the block has finished. If that happens, 
a segfault is exactly was it deserves IMHO.

Leaves the question, how stable this "interface" is?
Accessing _objects here belongs to voodoo programming practices of course, but 
the magic is locally limited to just two lines of code, which is acceptable in 
order to get this context manager working without messing with the rest of the 
code.

Opinions?

Thanks,
Pete
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[eryk sun]

On Thu, Jan 5, 2017 at 2:37 AM, Nick Coghlan  wrote:
> On 5 January 2017 at 10:28, Hans-Peter Jansen  wrote:
>> In order to get this working properly, the ctypes mapping needs a method to
>> free the mapping actively. E.g.:
>>
>> @contextmanager
>> def map_struct(m, n):
>> m.resize(n * mmap.PAGESIZE)
>> yield T.from_buffer(m)
>>  T.unmap_buffer(m)
>>
>> Other attempts with weakref and the like do not work due to the nature of the
>> ctypes types.
>
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:

I think that's the best you can do with the current state of ctypes.

from_buffer was made safer in Python 3 by ensuring it keeps a
memoryview reference in the _objects attribute (i.e.
CDataObject.b_objects). Hans-Peter's problem is a consequence of this
reference. Simply calling release() on the underlying memoryview is
unsafe. For example:

>>> b = bytearray(2**20)
>>> a = ctypes.c_char.from_buffer(b)
>>> a._objects

>>> a._objects.release()
>>> del b
>>> a.value
Segmentation fault (core dumped)

A release() method on ctypes objects could release the memoryview and
also clear the CDataObject b_ptr field. In this case, any function
that accesses b_ptr would have to be modified to raise a ValueError
for a NULL value. Currently ctypes assumes b_ptr is valid, so this
would require adding a lot of checks.

On a related note, ctypes objects aren't tracking the number of
exported views like they should. resize() should raise a BufferError
in the following example:

>>> b = (ctypes.c_char * (2**20))(255)
>>> m = memoryview(b).cast('B')
>>> m[0]
255
>>> ctypes.resize(b, 2**22)
>>> m[0]
Segmentation fault (core dumped)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Hi Nick,

On Donnerstag, 5. Januar 2017 12:37:20 Nick Coghlan wrote:
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:
> 
> class _T_data(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
> 
> class T:
> def __init__(self, buffer):
> self.data = _T_data.from_buffer(buffer)
> def close(self):
> self.data = None
> 
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> mapped = T(m)
> try:
> yield mapped
> finally:
> mapped.close()
> 
> Client code would then need to consistently access the struct through
> the data attribute:
> 
> with map_struct(m, 1) as a:
> a.data.foo = 1
> with map_struct(m, 2) as b:
> b.data.foo = 2

Thank you very much. Nice idea, indeed. 

Here's a slightly more complex example incorporating your idea:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx2-py

I'm not sure, if I like the resulting code more than the dreaded dels.
Real code based on this approach tends to be much more complex, and 
suffers appropriately.

Anyway, your solution is working fine, and provides a choice.
Much appreciated, Nick.

@ctypes developers: with an unmap operation available, we wouldn't need 
to go through these hoops, and ctypes powers would become even more 
accessible for such cool use cases ;)...

For now, I'm about to resign from using a context manager at all, since 
it uglifies the code in one way or another without buying much..

Cheers,
Pete
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes, memory mapped files and context manager

[Nick Coghlan]

On 5 January 2017 at 10:28, Hans-Peter Jansen  wrote:
> In order to get this working properly, the ctypes mapping needs a method to
> free the mapping actively. E.g.:
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> yield T.from_buffer(m)
>  T.unmap_buffer(m)
>
> Other attempts with weakref and the like do not work due to the nature of the
> ctypes types.

I don't know ctypes well enough myself to comment on the idea of
offering fully deterministic cleanup, but the closest you could get to
that without requiring a change to ctypes is to have the context
manager introduce a layer of indirection:

class _T_data(ctypes.Structure):
_fields = [("foo", ctypes.c_uint32)]

class T:
def __init__(self, buffer):
self.data = _T_data.from_buffer(buffer)
def close(self):
self.data = None

@contextmanager
def map_struct(m, n):
m.resize(n * mmap.PAGESIZE)
mapped = T(m)
try:
yield mapped
finally:
mapped.close()

Client code would then need to consistently access the struct through
the data attribute:

with map_struct(m, 1) as a:
a.data.foo = 1
with map_struct(m, 2) as b:
b.data.foo = 2

Something like http://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy
would let you make the indirection to a contained object transparent,
but as far as I can tell, wrapt doesn't currently support "closing" a
proxy by replacing the reference to the internal object with a
reference to None (adding that might be possible, but I don't
personally know wrapt well enough to guess the feasibility of doing
so).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com