[issue10746] ctypes c_long & c_bool have incorrect PEP-3118 type codes
Changes by Pauli Virtanen <p...@iki.fi>: -- pull_requests: +3286 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10746> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10746] ctypes c_long & c_bool have incorrect PEP-3118 type codes
Pauli Virtanen added the comment: Created backport PR for 3.6: https://github.com/python/cpython/pull/3241 Which versions take backports? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10746> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10746] ctypes c_long & c_bool have incorrect PEP-3118 type codes
Changes by Pauli Virtanen <p...@iki.fi>: -- pull_requests: +3284 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10746> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31276] PyObject_CallFinalizerFromDealloc is undocumented
New submission from Pauli Virtanen: It's unclear if PyObject_CallFinalizerFromDealloc is a public function or not. It is not documented, but it seems there's no other way to ensure that tp_finalize runs, at least for objects without Py_TPFLAGS_HAVE_GC. In the documentation of tp_finalize (https://docs.python.org/3/c-api/typeobj.html?highlight=tp_finalize#c.PyTypeObject.tp_finalize) there is the sentence: """It is called either from the garbage collector (if the instance is part of an isolated reference cycle) or just before the object is deallocated.""" However, it appears it is necessary to call it explicitly from any user-provided tp_dealloc. Indeed, there are several calls to PyObject_CallFinalizerFromDealloc in cpython/Modules/* e.g. in posixmodule.c:ScandirIterator_dealloc -- assignee: docs@python components: Documentation messages: 300842 nosy: docs@python, pv priority: normal severity: normal status: open title: PyObject_CallFinalizerFromDealloc is undocumented versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31276> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10746] ctypes c_long & c_bool have incorrect PEP-3118 type codes
Pauli Virtanen added the comment: Converted patch to Github PR + some cleanup. -- pull_requests: +61 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10746> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17706] Segfault in PyErr_SetObject
Pauli Virtanen added the comment: Yes, this is a bug in numpy.linalg --- the GIL is released but the error handling code assumes it's not. The error doesn't appear with in typical LAPACK installations, so this code branch was missed. -- nosy: +pv ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10744] ctypes arrays have incorrect buffer information (PEP-3118)
Pauli Virtanen p...@iki.fi added the comment:
The array notation is useful for arrays inside structs, such as T{(4)i(2,3)f}.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10744
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: The flags don't seem to be meant to describe the properties of the buffer, only what the exporter is required to fill in. STRIDES does not imply necessarily discontinuous, only that the `strides` field is present. The C_/F_/ANY_CONTIGUOUS flags imply that the memory layout of an n-dim array is C/Fortran/either contiguous. Why these flags imply STRIDES is probably to make the result unambiguous, and because typically when dealing with n-d arrays you usually need to know the strides anyway. `NULL` `strides` implies C-contiguous, so the CONTIG flag does not imply STRIDES (no idea why it's different from PyBUF_C_CONTIGUOUS). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: @skrah: Yes, Numpy exposes only a single buffer per object. Making this a requirement in the PEP would probably be a sane change, as there is probably little real-world need to allow it behave otherwise. Comment on the patch: it seems you do not track the re-export count in memory_getbuf: a = memoryview(obj) b = numpy.asarray(a) a.release() b[0] = 123 # -- BOOM: the buffer was already released Could be fixed by Py_INCREF(self-mbuf) in getbuffer and DECREF in releasebuffer. In this design, the only choice is to make the `release()` call to fail. (I had some code for n-dim slicing etc. in my first patch that could be useful to have too; I'll see if I find time to dig them out here.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: @skrah: Ahh, this always happens when I don't run it :) But my point stands -- the reason why it does not crash with Numpy is that Numpy calls PyMemoryView_FromObject to obtain a new memoryview and then uses PyMemoryView_GET_BUFFER. Along this code path the refcount of self-mbuf gets properly incremented, as it's explicitly done in PyMemoryView_FromObject. However, if you have a custom object Foo which just calls `PyObject_GetBuffer`, and then do the same sequence a = memoryview(whatever) b = Foo(a) # -- only grabs the buffer with PyObject_GetBuffer a.release() # -- will invalidate the buffer b.mogrify_buffer_contents() # -- boom Here, the buffer is released too early, as the refcount of `self-mbuf` is not incremented during the `PyObject_GetBuffer` call. Is there anything stopping us just storing the flags on PyManagedBuffer? Slicing memoryviews can invalidate the contiguity flags, and no-strides flags, so some checks are still probably needed in `memory_getbuf`. It's OK if the construction API requires the flag information in addition to the Py_buffer struct. This would be useful, yes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: skrah writes: I think slicing (esp. multidimensional slicing) would be greatly simplified if we added a requirement for the *exporting* object to provide a sliced view. (The same applies to sub-views, also see source comments below [1]). For example, an exporting object could provide a sliced view by adding a getslicedbufferproc to PyBufferProcs: int PyObject_GetSlicedBuffer(PyObject *obj, Py_buffer *view, int flags, PyObject *key); The same thing can be done via PyObject_GetBuffer(obj, view, flags); PyBuffer_Slice(view, sliced_view, flags, key); given an implementation of PyBuffer_Slice. The logic in PyBuffer_Slice does not depend on where the buffer comes from, and every buffer can be sliced. As far as I see, the advantage of `getslicedbufferproc` would be to make the implementation of PyMemoryView simpler, but not much else. In my view, having each exporter implement the same logic by itself would only be an unnecessary burden. o The invariant that all allocated memory in the buffer belongs to the exporting object remains intact. Numpy arrays do not have this invariant, and they happily re-export memory owned by someone else. This is one root of problems here: the PEP implicitly assumes that re-exporting buffers (e.g. memoryview's implementation of `getbuffer`) is done in the way Numpy does it. Because of this, there is no mechanism for incrementing the refcount of an existing buffer export. Maintaining the above invariant then unavoidably leads to strange behavior in corner cases (which probably are very rare, as mentioned above), and as happened here, make the implementation messy and lead to bugs. The invariant *is* required for guaranteeing that `memoryview.release()` always succeeds. Such a method probably wasn't foreseen in the PEP (and I did not remember that it existed in my first patch), as Numpy arrays don't have any equivalent. The alternatives here are (i) do as Numpy does and give up the invariant and allow `.release()` to fail in some cases, or (ii) document the corner cases in the interface spec and try to detect them and fail if they occur. Which of these is chosen probably does not matter much in practice, but having PyManagedBuffer will make implementing either choice easier. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: Lenard Lindstrom writes: Using Python reference counting and the garbage collector to control when PyBuffer_Release is called has problems. That's not what's being suggested. The refcounting discussed here is an implementation detail internal to memoryview, and does not affect the possibility to implement `release()` and context management reliably. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: Nick's plan of action above seems mostly OK to me. Though, it might be simpler to collapse the PyManagedBuffer object to PyMemoryView, even if this requires memoryviews to serve two purposes. [Nick:] I'm still not comfortable with a convention that relies on *clients* of the PEP 3118 API not mucking with the internals of the Py_buffer struct. Some points against: (i) Having to look up keys by memory address from an additional PyDict is more work for the exporter than just passing around some PyMem_Malloc'd information in `internal`. (ii) There is already an obj field in the structure that the consumers are supposed to not mess with. (iii) The exporter cannot use the `internal` field for anything if bf_releasebuffer cannot rely on it being intact. If the recommended consumer API is changed so that Py_buffer mainly sits inside a PyObject, it becomes more clear that Py_buffer is read-only for the consumer (-- which is what I believe the PEP intended, but did not write down). [Nick:] Altering release() to simply decrement the reference count of the managed buffer would defeat the whole point of having that method, so it may be necessary to allow early release with outstanding references and then include a still alive check in the code that allows access to the buffer details (similar to the way weak references work). Early release does not seem possible if the buffer does not come from the original object: lst = [] with memoryview(a) as m: b = numpy.array(m) lst.append(b) Now, m.__exit__ cannot release the buffer, since `b` holds a buffer-interface lock to `m`. `b` is 3rd party code, and does not know anything about MemoryViews. Some alternatives: (i) require that bf_getbuffer always gives a new lock on all exported buffers, if there are multiple, (ii) allow memoryview.__exit__ to fail silently, (iii) drop the context management. I guess (i) would be a sane choice -- I don't see many use cases for the same object exporting multiple different buffers. It's not needed by Numpy, and I suspect there is no existing 3rd party code that relies on this (because it doesn't work with the current implementation of memoryview :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: Hi, Please focus on the constraints of the consumer not mucking with the content of `Py_buffer`, and calling `bf_releasebuffer` only with data obtained from `bf_getbuffer` and only one. If we agree on this, then how to exactly to do the implementation is just a detail. The problem in the current way is that the structure sent to `bf_releasebuffer` does not contain the same data as what was filled in by `bf_getbuffer`, and since the contents are dup-ed, `bf_releasebuffer` is called multiple times with the same data. So, `bf_releasebuffer` cannot rely on (i) the data in Py_buffer being what `bf_getbuffer` put there, and (ii) getting the same Py_buffer data only once. So, `bf_releasebuffer` cannot be used to release any resources allocated in `bf_getbuffer`. For example, Numpy's PEP 3118 implementation does not define a `bf_releasebuffer` although it needs to allocate resources in each call to `bf_getbuffer`. Instead, it has to keep a track of allocated memory, and deallocate all of them on `array_dealloc`. This is a PITA, and one that could be avoided by a small change in the PEP implementation and documentation in Python. Some worrying things here: * memoryview_getbuffer() doesn't call the original object's getbuffer. This means that if I do: m = memoryview(some_object) n = memoryview(m) m.release() n ends up holding a buffer to some_object's memory, but some_object doesn't know about it and can free the pointer at any time. Good point. There are two possible solutions to this: - Keep a count of how many buffers memoryview() has exported, and do not allow memoryview.release() if some are active. In a sense, this would be more in line with the PEP: the PyMemoryViewObject would here act as an ordinary object exporting some block of memory, and not do any magic tricks. It would guarantee that the buffers it has exported stay valid. - Add additional fields to `PyMemoryViewObject` for storing new `strides`, `format`, and `shape` that override the stuff in Py_buffer. This would allow for calling `PyObject_GetBuffer` for a second time. Both would be reasonably easy to do. *** Calling PyObject_GetBuffer to get a new hold of a buffer needs some precautions, though. For example: mem = memoryview(some_object) # do some stuff mem2 = memoryview(some_object) assert mem.format == mem2.format # not guaranteed so there is some extra work `memoryview_getbuffer` could do on top of calling PyObject_GetBuffer. * same for _get_sub_buffer_index() and _get_sub_buffer_slice0(). Actually, the whole concept of non-owner memoryviews seems flawed. If the parent memoryview keeps its the memory valid as long as such sub-memoryviews exist, such concerns go away. * the fact that slicing increments the parent memoryview's refcount means that a loop like: while len(m): # do something m = m[1:] will end up keeping N chained memoryviews on the heap. Currently only the last memoryview remains alive. This has an easy solution: if the parent is a non-owner memoryview, take a reference to its obj. This then keeps only buffer-owning view on the stack. Some other things: * why do you accept the ellipsis when indexing? what is it supposed to mean? Same meaning as in Numpy. a[...] == a * please don't use #warning. Just put the TODO in a /* ... */. Sure. * please no #if PY_VERSION_HEX = 0x0300. Just make your source py3k-compatible and we'll take care of backporting it to 2.x if your patch is accepted ;) Those are just so that the backporting would be easy. It's not on the stage of a serious patch yet :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment:
Hmm, there's a misunderstanding. bf_releasebuffer is called exactly
once for each call to bf_getbuffer.
Wrong: http://bugs.python.org/issue7433
static int
memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags)
{
int res = 0;
CHECK_RELEASED_INT(self);
if (self-view.obj != NULL)
res = PyObject_GetBuffer(self-view.obj, view, flags);
if (view)
dup_buffer(view, self-view);
return res;
}
After this, PyBuffer_Release will be called twice: once on the data in *view,
by whoever acquired the buffer from memoryview, and once on self-view, by
memory_dealloc. Both with the same bit-by-bit content of the Py_buffer
structure.
Because there are two Py_buffer structures here, setting view.obj to NULL in
PyBuffer_Release does not guarantee correct calls to bf_releasebuffer.
Note that the view.internal pointer is also clobbered above.
So, `bf_releasebuffer` cannot rely on (i) the data in Py_buffer
being what `bf_getbuffer` put there,
Well, why should it rely on that?
Because that makes implementing the exporter much easier. Also, writing an
implementation for MemoryViewObject does not require clobbering the structure,
and I doubt it helps much.
So, `bf_releasebuffer` cannot be used to release any resources
allocated in `bf_getbuffer`.
AFAICT, it can. That's what the internal pointer is for.
Sure, guaranteeing that view-internal pointer is not toyed with would also be
enough.
But the documentation should spell out very explicitly what the
bf_releasebuffer call can rely on.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10181
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: [clip] This is a different issue. It is the issue relevant for this discussion. As written in my comment: So, `bf_releasebuffer` cannot rely on (i) the data in Py_buffer being what `bf_getbuffer` put there, and (ii) getting the same Py_buffer data only once. PyObject_GetBuffer() is called twice too: once when creating the memoryview, once when calling memory_getbuf. So again, bf_getbuffer is called the same number of times as bf_releasebuffer. Yes, it is called twice, but with exactly the same data in Py_buffer. So I would rather say that bf_releasebuffer is called twice on the Py_buffer returned by the first Getbuffer, and zero times for the buffer returned by the second one. Note that the view.internal pointer is also clobbered above. Are you sure? memoryobject.c doesn't touch that pointer at all. dup_buffer does *dst = *src, which overwrites the view.internal pointer obtained from one GetBuffer call with a pointer obtained from a previous one. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: Ok, good, that diversion was then avoided :) *** So, am I on the right track that there would not be many objections to clarifying the docs of Py_buffer spec by stating: - For each buffer yielded by `bf_getbuffer`, `bf_releasebuffer` is called exactly once. Each `bf_releasebuffer` call is guaranteed to get the same view-internal pointer as filled in previously by the corresponding `bf_getbuffer`. All other fields in `Py_buffer` may be modified by the consumer, and `bf_releasebuffer` may not assume they contain valid data. - The exporter of the buffer must ensure that apart from the contents of the memory pointed to by `buf`, the contents of the returned Py_buffer (format, strides, shape, etc.) remain unchanged. If the latter is not true, it will lead to no end of trouble in e.g. multithreaded programs. What about the more strict requirement: - `bf_releasebuffer` is guaranteed that all fields except `obj` are not modified? This could simplify implementation of exporters, and I suspect that MemoryViewObject is a special case, as most consumers probably would not want to make such modifications. I can try preparing an updated patch based on some combination of the above, taking into account your comments. BTW, should I take this discussion to Python-dev? So far, I kept it here, as this bug report seemed to be about general issues in the current implementation and spec. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: Ok, here's a patch with some suggested documentation changes + the minimal changes in memoryview to make bf_releasebuffer behave as advertised. Probably shouldn't be applied as-is, though. Problems remain. Suppose `bf_getbuffer` for memoryview is implemented so that it returns a view exported by the original object (and not the memoryview). Consider an example: obj = PictureSet() # exposes a buffer, say, to pic_1 a = memoryview(obj) # points to pic_1 Here, `a` grabs a lock to pic_1. obj.choose_picture(2)# switches the exposed buffer to pic_2 b = memoryview(obj) # points to pic_2 c = memoryview(a) Here, PyObject_GetBuffer in bf_getbuffer for `a` grabs a lock to pic_2 and passes it on to `c`. The spec specifies no ways to get additional locks on pic_1. a.release() Now lock on pic_1 is released, and the buffer may be freed (also, if the handling of shape/stride arrays in memoryview is as it is now, also those are invalidated). One of the two is now true: `memoryview(a)` does not always describe the same area of memory as `a`, OR, `c` ends up with a lock on the wrong area of memory and the program hits a SEGV. [clip] In a sense, this would be more in line with the PEP: the PyMemoryViewObject would here act as an ordinary object exporting some block of memory, and not do any magic tricks. Well, that sounds wrong to me. The memoryview doesn't export anything; the original object does. Having the memoryview own the exported buffer would be a simple solution to the above issue. The alternative would be to adjust the spec so that the above type of behavior is forbidden (any buffers and the shape/stride arrays ever exported must remain valid until the last one is released). Not sure if it makes sense to do this if the only gain is a simplification in the implementation of memoryview. It would guarantee that the buffers it has exported stay valid. How would it, since it doesn't know the original object's semantics? The original object must guarantee that the buffers remain valid until released, as specified in the PEP. Of course, this requires that memoryview counts how many buffers it has exported, and allows release() only if the count is zero. -- Added file: http://bugs.python.org/file20756/buffer-interface-clarify.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)
Pauli Virtanen p...@iki.fi added the comment: I spent today some time to rewrite `memoryobject.c`, and cleaning up the Py_buffer handling in it. (I wrote also the Numpy PEP 3118 implementation, so this was straightforward to do.) The end result is here: https://bitbucket.org/pv/cpython-stuff/changesets (bookmark bug/memoryview) Some points of note: - Clarification of ownership conventions for Py_buffer in docs: http://bitbucket.org/pv/cpython-stuff/changeset/805971191369 The new implementation is written to respect the invariants mentioned there. - Rewritten memoryobject: http://bitbucket.org/pv/cpython-stuff/src/817edc63ce4d/Objects/memoryobject.c I didn't yet heavily test this (eg. against Numpy), but at least the Python test suite passes. There are also some minor corners that need to be polished. Also some new tests would be needed, as I slightly extended the slicing capabilities of memoryview. Only one nasty point turned up: the existence of PyMemoryView_FromBuffer. This function breaks the ownership rules: the pointers in Py_buffer structure can point inside the structure --- and the structure can reside e.g. on the stack. Deep copying Py_buffer cannot in general be done, because this can mess up whatever bf_releasebuffer tries to do. The workaround here was to do a deep copy *only* for those pointers that point inside the struct --- although this violates the spirit of the ownership model, it should be essentially always safe to do, and I believe it is the correct solution (at least if PyMemoryView_FromBuffer is to be retained). *** Comments are welcome. -- nosy: +pv ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10744] ctypes arrays have incorrect buffer information (PEP-3118)
New submission from Pauli Virtanen p...@iki.fi: Ctypes arrays have invalid buffer interface information (on Python 3.1.2): import ctypes x = (ctypes.c_double*2)() y = memoryview(x) y.shape (2,) y.format '(2)d' This implies that the array contains 2 items, each consisting of 2 floats, which is incorrect -- the shape information is duplicated. A suggested fix is attached. (From http://projects.scipy.org/numpy/ticket/1699) -- assignee: theller components: ctypes files: 001-ctypes-fix-pep-3118-format-strings-for-arrays.patch keywords: patch messages: 124406 nosy: pv, theller priority: normal severity: normal status: open title: ctypes arrays have incorrect buffer information (PEP-3118) type: behavior versions: Python 3.1, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file20123/001-ctypes-fix-pep-3118-format-strings-for-arrays.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10744 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10746] ctypes c_long c_bool have incorrect PEP-3118 type codes
New submission from Pauli Virtanen p...@iki.fi:
Currently on Python 3.x:
import ctypes
memoryview(ctypes.c_long()).format
'l'
This is invalid on 64-bit platforms: the above means 32-bit little-endian
float. The '' endian specification turns on the standard size mode
(similarly as for the struct module), which makes type character have a
platform-independent meaning.
Unfortunately, the struct module format syntax does *not* allow specifying
native-size non-native-endian items. So just replacing '' by '^' cannot be in
general be done.
Suggested fix attached. It adds a converter function that maps the
platform-dependent ctypes codes to struct module standard-size codes; handling
c_long and c_bool specially.
***
After this patch (and the one in http://bugs.python.org/issue10744 ):
import numpy as np
from ctypes import *
class Point(Structure):
... _fields_ = [(x, c_long), (y, c_long)]
...
class StructWithArrays(Structure):
... _fields_ = [(x, c_long * 3 * 2), (y, Point * 4)]
...
x = StructWithArrays()
y = np.asarray(x)
y.dtype
dtype([('x', 'i8', (2, 3)), ('y', [('x', 'i8'), ('y', 'i8')], (4,))])
y['x'] = [[1,2,3],[4,5,6]]
y['y']['x'] = np.arange(4) + 10
y['y']['y'] = np.arange(4) + 20
x.x[0][0]
1
x.x[0][1]
2
x.x[0][2]
3
x.y[0].x
10
x.y[1].x
11
x.y[0].y
20
x.y[1].y
21
--
files: 001-ctypes-fix-pep-3118-type-codes-for-c-long-and-c-bool.patch
keywords: patch
messages: 124411
nosy: pv
priority: normal
severity: normal
status: open
title: ctypes c_long c_bool have incorrect PEP-3118 type codes
Added file:
http://bugs.python.org/file20124/001-ctypes-fix-pep-3118-type-codes-for-c-long-and-c-bool.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10746
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10746] ctypes c_long c_bool have incorrect PEP-3118 type codes
Changes by Pauli Virtanen p...@iki.fi: -- assignee: - theller components: +ctypes nosy: +theller type: - behavior versions: +Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10746 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3132] implement PEP 3118 struct changes
Pauli Virtanen p...@iki.fi added the comment: I still like the idea of scoped endianness markers in the substructs, but if we have to abandon that for compatibility with NumPy that's okay. That, or change the Numpy implementation. I don't believe there's yet much code in the wild that changes the alignment specifier on the fly. [clip: 'O' format code] So the object returned by 'pack' would somehow have to be something other than a plain string, so that it can deal with automatically doing the DECREF of the held PyObject* pointers when it goes out of scope. Yes, the packed object would need to own the references, and it would be the responsibility of the provider of the buffer to ensure that the pointers are valid. It seems that it's not possible for the `struct` module to correctly implement packing for the 'O' format. Unpacking could be possible, though (but then if you don't have packing, how write tests for it?). Another possibility is to implement the 'O' format unsafely and leave managing the reference counting to whoever uses the `struct` module's capabilities. (And maybe return ctypes pointers on unpacking.) [clip] What's the need to have the 'O' format in the struct module? Is it really necessary there? Can we get away with not implementing it? Numpy arrays, when containing Python objects, function as per the 'O' format. However, for the struct module, I don't see what would be the use case for the 'O' format. BTW, does this already exist in a released version of NumPy? If not, when is it likely to appear in the wild? It's included since the 1.5.0 release which came out last July. *** I think after the implementation is done, the PEP probably needs to be amended with clarifications (and possibly cutting out what is not really needed). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3132] implement PEP 3118 struct changes
Pauli Virtanen p...@iki.fi added the comment: For reference, Numpy's PEP 3118 implementation is here: http://github.com/numpy/numpy/blob/master/numpy/core/_internal.py#L357 http://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/buffer.c#L76 It would be a good idea to ensure that the numpy and struct implementations are in agreement about details of the format strings. (I wouldn't take the Numpy implementation as the definitive one, though.) - The sub-structs in Numpy arrays (in align=True mode) are aligned according to the maximum alignment of the fields. - I assumed the 'O' format in the PEP is supposed to be similar to Numpy object arrays. This implies some reference counting semantics. The Numpy PEP 3118 implementation assumes the memory contains borrowed references, valid at least until the buffer is released. Unpacking 'O' should probably INCREF whatever PyObject* pointer is there. - I assumed the alignment specifiers were unscoped. I'm not sure however whether this is the best thing to do. - The function pointers and pointers to pointers were not implemented. (Numpy cannot represent those as data types.) -- nosy: +pv ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9838] Inadequate C-API to Python 3 I/O objects
New submission from Pauli Virtanen p...@iki.fi: The C-API exposed by the `io` module on Python 3.1/3.2 is very limited, and makes interfacing with Python file objects in extension modules difficult. In more detail: 1) Because the Python layer has buffering etc., the file handle returned by `PyObject_AsFileDescriptor` is not usable as-is. It requires flush and seek before use, every time there is a chance that the file object has been accessed on the Python side. 2) There are no C-API functions such as the minimal set of `PyFile_Write(buf, length)`, `PyFile_Read(buf, length)`, `PyFile_Seek(pos, whence)`, `PyFile_Tell()`. Instead, every call must go through PyObject_CallMethod, and the file objects only handle `PyBytes` and `PyByteArray` which are cumbersome and inefficient to use in extension modules. -- components: Extension Modules messages: 116188 nosy: pv priority: normal severity: normal status: open title: Inadequate C-API to Python 3 I/O objects type: feature request versions: Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9838 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7433] MemoryView memory_getbuf causes segfaults, double call to tp_releasebuffer
Pauli Virtanen p...@iki.fi added the comment: Why do you say that: There is no feasible way the bf_releasebuffer can keep track of how many calls to it have been made. I was probably thinking about allocating new temporary arrays for strides etc. on each *_getbuffer -- if that's done, then manually keeping track of all the allocated memory seems like a waste of effort (ie. not feasible). But yes, if memory allocated for entries in Py_buffer is shared between all exported buffer views, that sounds better -- for some reason I didn't think about that... So we'll do it like this in Numpy then. But still, I take it that the way it currently works is not the intended behavior? The segmentation faults caused by this came as a bit of a surprise to me, as the assumption about paired *_getbuffer and *_releasebuffer calls is very natural. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7433 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7433] MemoryView memory_getbuf causes segfaults, double call to tp_releasebuffer
Pauli Virtanen p...@iki.fi added the comment: I think this is an implementation issue in MemoryView rather than an issue with the buffer interface. PEP 3118 states, This same bufferinfo structure must be passed to bf_releasebuffer (if available) when the consumer is done with the memory. -- this is not guaranteed by the current MemoryView implementation. The calls are not paired: the *_getbuf calls fill in structures with data view1, and view2. The *_releasebuf calls receive structures with data view1, and view1. The data filled in the second getbuf call (view2) is never passed back to *_releasebuf, as it is overwritten with view1 data by dup_buffer. To work around this, *_releasebuf must be written so that it does not use the view pointer passed to it -- the data structure may have been shallow copied and any memory pointers in it may have already been freed. I can try to cook up a patch fixing this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7433 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7433] MemoryView memory_getbuf causes segfaults, double call to tp_releasebuffer
New submission from Pauli Virtanen p...@iki.fi:
The following code causes a segmentation fault (or glibc error, or other
problems):
x = someobject()
y = memoryview(x)
z = memoryview(y)
The problem is that someobject.bf_releasebuffer will be called two times
with an identical Py_buffer structure.
This can be seen in memoryobject.c:
static int
memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags)
{
int res = 0;
/* XXX for whatever reason fixing the flags seems necessary */
if (self-view.readonly)
flags = ~PyBUF_WRITABLE;
if (self-view.obj != NULL)
res = PyObject_GetBuffer(self-view.obj, view, flags);
if (view)
dup_buffer(view, self-view);
return res;
}
At the end of the call, view and self-view contain identical data
because of the call to dup_buffer.
static void
memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view)
{
PyBuffer_Release(view);
}
But when the outer memoryview is destroyed, memory_releasebuf calls
PyBuffer_Release for the original object once.
And when the inner memoryview is destroyed, PyBuffer_Release is called
by memory_dealloc the second time. Both calls supply an identical
Py_buffer structure.
Now, if the original object's bf_getbuffer and bf_releasebuffer allocate
some memory dynamically, this will likely cause a double-free of memory,
usually leading to a segmentation fault.
There is no feasible way the bf_releasebuffer can keep track of how many
calls to it have been made. So probably the code in memory_getbuf is
wrong -- at least the dup_buffer function looks wrong.
--
components: Interpreter Core
messages: 95952
nosy: pv
severity: normal
status: open
title: MemoryView memory_getbuf causes segfaults, double call to
tp_releasebuffer
versions: Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7433
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7433] MemoryView memory_getbuf causes segfaults, double call to tp_releasebuffer
Changes by Pauli Virtanen p...@iki.fi: -- type: - crash ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7433 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7385] MemoryView_FromObject crashes if PyBuffer_GetBuffer fails
Changes by Pauli Virtanen p...@iki.fi: -- type: - crash ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7385 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7385] MemoryView_FromObject crashes if PyBuffer_GetBuffer fails
New submission from Pauli Virtanen p...@iki.fi:
In Objects/memoryobject.c:PyMemoryView_FromObject there's a
_PyObject_GC_UNTRACK unpaired with corresponding _PyObject_GC_TRACK,
which seems to cause a segmentation fault. This can be triggered by
calling PyMemoryView_FromObject on an object whose bf_getbuffer returns
an error.
PyMemoryView_FromObject(PyObject *base) {
...
if (PyObject_GetBuffer(base, (mview-view), PyBUF_FULL_RO) 0) {
Py_DECREF(mview);
return NULL;
}
...
_PyObject_GC_TRACK(mview);
}
...
static void memory_dealloc(PyMemoryViewObject *self) {
_PyObject_GC_UNTRACK(self);
}
--
components: Interpreter Core
messages: 95660
nosy: pv
severity: normal
status: open
title: MemoryView_FromObject crashes if PyBuffer_GetBuffer fails
versions: Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7385
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3655] latexwriter: \footnotemark can only take numbers as arguments
New submission from Pauli Virtanen [EMAIL PROTECTED]: LaTeXTranslator.visit_footnote_reference generates improper Latex if the footnote marker given is not a number. This will result in a Latex error if the RST document contains footnotes where the marker is not a number. The problem is that the Latex commands \footnotemark and \footnotetext apparently expect their [] argument to always be a number. For example: \footnotemark[x] results to error: ! Missing number, treated as zero. to be read again x l.4733 ...\footnotemark[x] -- assignee: georg.brandl components: Documentation tools (Sphinx) messages: 71817 nosy: georg.brandl, pv severity: normal status: open title: latexwriter: \footnotemark can only take numbers as arguments ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3655 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3422] sphinx.doc.autodoc: Hook for changing argspec
Pauli Virtanen [EMAIL PROTECTED] added the comment: Thanks, works OK. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3422] sphinx.doc.autodoc: Hook for changing argspec
Pauli Virtanen [EMAIL PROTECTED] added the comment: Suggested patch attached. Tested on Numpy documentation. It adds a new signal, autodoc-process-signature(app, what, name, obj, options, signature, return_annotation) which is assumed to return either None or a new (signature, return_annotation) tuple. Warnings are raised only if introspection fails and none of the event handlers returns a new signature. -- keywords: +patch Added file: http://bugs.python.org/file10971/autodoc-process-signature.patch ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3422] sphinx.doc.autodoc: Hook for changing argspec
New submission from Pauli Virtanen [EMAIL PROTECTED]: It would be useful if the autodoc-process-docstring event from sphinx.ext.autodoc allowed to change the argspec of the function being documented. Some other hook for changing the function signature would also do. We are using Sphinx for generating a reference guide for Numpy, where many of the functions are from extension modules for which inspect.getargspec does not work. Instead, the function signature is contained within the object's docstring. Right now I'm simply monkeypatching sphinx.ext.autodoc.format_signature, but a cleaner approach would be better. It seems that this would be fairly easy to implement in generate_rst. Perhaps a .signature attribute to the Options passed to the hook would be an acceptable solution? I can write a patch doing this, if someone doesn't do this faster. -- assignee: georg.brandl components: Documentation tools (Sphinx) messages: 70096 nosy: georg.brandl, pv severity: normal status: open title: sphinx.doc.autodoc: Hook for changing argspec type: feature request versions: 3rd party ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3422 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
