[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-04-09 Thread Josh Triplett
Josh Triplett j...@joshtriplett.org added the comment: I currently use Python 2.7, and I'd like to make use of memoryview. Specifically, I work on BITS (http://biosbits.org/), which runs Python in ring 0 as part of GRUB, and I'd like to use memoryview to give Python access to data in

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-04-09 Thread Josh Triplett
Josh Triplett j...@joshtriplett.org added the comment: I currently use Python 2.7, and I'd like to make use of memoryview. Specifically, I work on BITS (http://biosbits.org/), which runs Python in ring 0 as part of GRUB, and I'd like to use memoryview to give Python access to data in

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-03-05 Thread Roundup Robot
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 373f6cdc6d08 by Stefan Krah in branch 'default': Issue #10181: The decision was to raise a buffer error in memory_exit() http://hg.python.org/cpython/rev/373f6cdc6d08 --

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-03-02 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: It occurs to me that one thing that *could* be backported to early versions are some of the documentation improvements on how to correctly handle the lifecycle of fields in Py_buffer. That gets messy though because memoryview doesn't behave

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-03-01 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: From the PEP: The buffer interface (type Py_buffer, type slots bf_getbuffer and bf_releasebuffer, etc) has been omitted from the ABI, since the stability of the Py_buffer structure is not clear at this time. Inclusion in the ABI can

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-29 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I'm busy adding the C-API changes to the docs. Regarding the stable ABI: The general mood was to *keep* the removal of the buffer interface for some time, did I get that right? In that case this removal (especially of the Py_buffer

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-29 Thread Roundup Robot
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 9d3c7dd55c7f by Stefan Krah in branch 'default': Issue #10181: Add warning that structure layouts in memoryobject.h and http://hg.python.org/cpython/rev/9d3c7dd55c7f --

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-29 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: For 3.2, there's no removal to document - we asked MvL to drop the buffer APIs from PEP 384, so they've never been part of the stable ABI. From the PEP: The buffer interface (type Py_buffer, type slots bf_getbuffer and bf_releasebuffer, etc)

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-26 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick Coghlan rep...@bugs.python.org wrote: PEP section makes sense - I plan to mark PEP 3118 as Final once you commit this (or you can do that yourself, for that matter). Array features are complete except for multi-dimensional

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-26 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Aw, I guess I was too optimistic in thinking this was the last gap before we could declare it Final... +1 on creating a new feature request for NumPy style multi-dimensional indexing and slicing support on memoryview (I'm assuming that's what

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-25 Thread Roundup Robot
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 3f9b3b6f7ff0 by Stefan Krah in branch 'default': - Issue #10181: New memoryview implementation fixes multiple ownership http://hg.python.org/cpython/rev/3f9b3b6f7ff0 -- nosy: +python-dev

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-24 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Thanks, Nick. I'll try to get it done this weekend. I've uploaded Misc/NEWS and Doc/whatsnew/3.3.rst (my apologies to Antoine for plagiarizing the first sentence, I found it hard to come up with a better version). I wasn't sure

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-24 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: PEP section makes sense - I plan to mark PEP 3118 as Final once you commit this (or you can do that yourself, for that matter). -- ___ Python tracker rep...@bugs.python.org

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-02-21 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Latest version looks good to me - I vote for landing it whenever you're ready :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-31 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file24381/a5c4a96dc981.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-31 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I've uploaded a new patch that should address the remaining issues: o In the documentation _testbuffer has been replaced by m.cast() + the now multi-dimensional m.tolist(). o I restored the state of the limited API. If we

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-29 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Antoine Pitrou rep...@bugs.python.org wrote: I thought the whole Py_buffer API was only temporarily removed from the limited API until the issues were sorted out: http://bugs.python.org/issue10181#msg125462 I'm talking about

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-29 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Of course this could also be exposed as a function, e.g.: /* stealing a reference to bytes */ PyMemoryView_FromBytesAndInfo(PyObject *bytes, Py_buffer *info); I think we should minimize the number of reference-stealing functions. So

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I've been busy fixing the memoryview.h issues first. Could you take a look at: http://hg.python.org/features/pep-3118/file/f020716704d4/Include/memoryobject.h Changes: a) Make all functions and the two buffer access macros part

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick Coghlan rep...@bugs.python.org wrote: Radical suggestion: make it public as collections.simple_ndarray? Heh, that's a nice one. :) While I think that the code in _testbuffer.c is sound and very well tested, the API is low-level

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Nick Coghlan rep...@bugs.python.org wrote: Radical suggestion: make it public as collections.simple_ndarray? Heh, that's a nice one. :) Well, the question is: does it have a point? Is this ndarray useful outside of any third-party

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Antoine Pitrou rep...@bugs.python.org wrote: Nick Coghlan rep...@bugs.python.org wrote: Radical suggestion: make it public as collections.simple_ndarray? Heh, that's a nice one. :) Well, the question is: does it have a point?

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: a) Make all functions and the two buffer access macros part of the limited API again. Hmm, I don't think buffer access macros should be part of the limited API. If they are truely important (which I doubt), we should have equivalent

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Antoine Pitrou rep...@bugs.python.org wrote: a) Make all functions and the two buffer access macros part of the limited API again. Hmm, I don't think buffer access macros should be part of the limited API. If they are

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Antoine Pitrou rep...@bugs.python.org wrote: a) Make all functions and the two buffer access macros part of the limited API again. Hmm, I don't think buffer access macros should be part of the limited API. If they are truely

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: I'm with Antoine here - we want to be *very* conservative with what we expose through the limited API. Due to the ABI compatibility promise, anything exposed that way is very hard to change. Keeping things out of the limited API isn't really

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-28 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Also, +1 to replacing _testbuffer with .cast() to demonstrate the multi-dimensional support. Without an actual multi-dimensional array object in the standard library, demonstrating those aspects is always going to be a little tricky. However,

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-27 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Aside from some minor comments that I included in my review, the latest patch gets a +1 from me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-27 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Antoine's review picked up on several issues I missed or glossed over - I actually agree with his point about making most of the new APIs private rather than public. With regards to exposing _testbuffer in the documentation of memoryview's

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-27 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: With regards to exposing _testbuffer in the documentation of memoryview's hash support, perhaps it would be better to use a 1D bytes object + memoryview.cast() to get an officially supported multi-dimensional view of a region of memory? By

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-27 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I've added some comments on Rietveld. This time, I pasted the email addresses manually into the CC field, apparently without success (I didn't receive mail). Regarding the use of _testbuffer in the docs: I agree that it's strange, on the

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-27 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Radical suggestion: make it public as collections.simple_ndarray? (prefixing with simple_ to be explicit that this is not even *close* to being the all-singing, all-dancing NumPy.ndarray) -- ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-25 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file24323/8dd9f0ea4876.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-18 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Revisiting memoryview.size: I foresee problems for NumPy users, since array.size has a different meaning there: x = array([[1,2,3], [4,5,6]], dtype='q') x.shape (2, 3) x.itemsize 8 len(x) 2 x.size 6 x.nbytes 48 So here we have:

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-18 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: nbytes sounds reasonable to me, given the unfortunate ambiguity of both size and len. As far as #12834 goes, I'm happy to go along with whatever you think is best. You've spent a lot more time down in the guts of the implementation than I

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-18 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Err, make that #12384 (oops) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-18 Thread Nick Coghlan
Changes by Nick Coghlan ncogh...@gmail.com: -- Removed message: http://bugs.python.org/msg151539 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-17 Thread Kristján Valur Jónsson
Kristján Valur Jónsson krist...@ccpgames.com added the comment: Thanks Nick. Looking through this discussion it looks as though you encountered all the confusing bits that I ran into (dup_buffer, ownership issues, reference counting and so forth.) Good work. --

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-17 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Thanks for the comments. Most of them should be easy to fix. Nick Coghlan rep...@bugs.python.org wrote: [...] expose the Py_buffer len field as memoryview.size instead of memoryview.len (to reduce confusion with len(memoryview) and to

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2012-01-16 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Finally reviewed Stefan's latest patch. It mostly looks great to me. Aside from a few minor cosmetic quibbles, the only substantive change I would like is to expose the Py_buffer len field as memoryview.size instead of memoryview.len (to

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-11-13 Thread Paul Moore
Changes by Paul Moore p.f.mo...@gmail.com: -- nosy: +pmoore ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list mailing

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-09-18 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file23185/4492afe05a07.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-09-18 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Revision 4492afe05a07 allows memoryview to handle objects with an __index__() method. This is for compatibility with the struct module (See also #8300). -- ___ Python tracker

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-09-08 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file23119/51cedae5acfc.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-09-08 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Here's a completely restructured memoryview implementation that I believe is quite robust. Both memoryobject.c (the parts I worked on, which is 91%) and _testbuffer.c have 100% code coverage, including all error conditions [1].

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-09-08 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: is over 1000x slower that the optimized versions of memoryview. Oh dear, scratch that. Lets make that 15x. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-08-16 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file22909/70a8ccd53ade.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-08-16 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: 70a8ccd53ade fixes conversion of NumPy-style arrays to a suboffset representation. The previous representation did not work for slicing non-contiguous arrays with backward strides. Also, I've added long comments that hopefully explain

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-08-10 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file22873/78fb1181787a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-08-10 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I thought it might be productive to switch to documentation/test driven development for PEP-3118 in general. So I updated the documentation, trying to spell out the responsibilities of both exporter and consumer as clearly as possible.

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-08-09 Thread Lenard Lindstrom
Changes by Lenard Lindstrom le...@telus.net: -- nosy: -kermode ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___ Python-bugs-list

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-06 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Antoine is right, this needs to be fixed. I think that for *practical* purposes, the existing release() method already behaves like a tryrelease() method: Traceback (most recent call last): File stdin, line 1, in module BufferError:

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-06 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: [The first part of the message again, this time via the web interface.] Antoine is right, this needs to be fixed. I think that for *practical* purposes, the existing release() method already behaves like a tryrelease() method: b =

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-06 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Le mercredi 06 juillet 2011 à 12:39 +, Stefan Krah a écrit : Antoine, was this roughly your suggestion? I think so (!), but I also agree with Nick that raising BufferError when calling release() on a memoryview with exports is a reasonable

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-06 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Yeah, the reason my originally proposed semantics were wrong is because copying (or slicing) a memoryview object and then explicitly releasing that object would always fail through no fault of that code. That would be broken and the only way

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick, Pauli, thanks for all the comments. I'm busy implementing the easy changes; then it'll be easier to deal with the flags issues. Pauli: Does numpy use the (undocumented) smalltable array in the Py_buffer structure? We would

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file22577/718801740bde.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I've uploaded a revised version that addresses several suggestions. I think we have agreement on those now: - Officially ditch smalltable. - Comment static storage fields inside PyMemoryViewObject. - Improve refcounting in

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: I'm slightly confused about the implication chain in the flags. PyBUF_STRIDES seem to allow for discontiguous arrays, yet STRIDES - ND - C_CONTIGUOUS. PyBUF_FULL[_RO]

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: I'm slightly confused about the implication chain in the flags. PyBUF_STRIDES seem to allow for discontiguous arrays, yet STRIDES - ND - C_CONTIGUOUS. To be honest I have never understood anything about these flags, and I doubt anyone without

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Pauli Virtanen
Pauli Virtanen p...@iki.fi added the comment: The flags don't seem to be meant to describe the properties of the buffer, only what the exporter is required to fill in. STRIDES does not imply necessarily discontinuous, only that the `strides` field is present. The C_/F_/ANY_CONTIGUOUS flags

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: It took me a bit of thinking, but I figured out why the contiguous flags imply STRIDES. A quick recap of all the flags: WRITABLE - error if can't support write access FORMAT - request format info in Py_buffer struct. Should never error, but

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: At least, that's the explanation based on the PEP - not sure where CONTIG as an alias for ND (N-dimensional) comes from. But then, smalltable was an undocumented novelty, too :) -- ___ Python

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: To address the should PyManagedBuffer be public? question: yes, I think so. Given the amount of grief the raw PEP 3118 API has caused the memoryview implementation, I expect the easier lifecycle management provided by the PyObject based API

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Regarding the Reitveld cc field: I tend not to add anyone to that and instead post comments to the tracker item to say that I've finished a review in Reitveld. If people want to see details they can go look at the review itself (or remove

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Regarding the Reitveld cc field: I tend not to add anyone to that and instead post comments to the tracker item to say that I've finished a review in Reitveld. If people want to see details they can go look at the review itself (or remove

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: I don't think that's a bug, it's a missing feature in the integration (there's a request on the metatracker to add automatic notifications of new reviews on the bug itself). I did mention the review above but it would have been easy to miss

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: I don't think that's a bug, it's a missing feature in the integration (there's a request on the metatracker to add automatic notifications of new reviews on the bug itself). It is a bug, actually. People on the nosy list are also on the

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: (and so, for the record, I've added my own small review :)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Moving this discussion out of the review comments: Antoine is wanting to make release() nondeterministic by having the underlying buffer only released when all views using it either have release() called or are no longer referenced. I

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Antoine is wanting to make release() nondeterministic by having the underlying buffer only released when all views using it either have release() called or are no longer referenced. That's not nondeterministic if everyone calls release(). Less

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Antoine Pitrou rep...@bugs.python.org wrote: My issue is that until now sliced memoryviews are independent objects and are not affected by the releasing of the original memoryview. With this patch, they are, and that's why I'm

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: I thought the rationale for the release() method was to allow sequences like: b = bytearray() m1 = memoryview(b) m1.release() - must call releasebuffer instantly. b.resize(10) - this might fail otherwise if the garbage collection is too

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: If someone is calling release() on all of their views (including slices) than they won't have any problems. The only way they can get into trouble is if they have a slice or copy that they *aren't* explicitly releasing, and in that case they

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Oops, Antoine's right, the release() semantics in the patch are broken, albeit for the precisely opposite reasons: that example will actually blow up with BufferError inside some_library_function(). I withdraw my objection - Antoine's right

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-05 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Sorry, I mischaracterised Antoine's suggestion in my last comment. It's more in the nature of ManagedBuffer exposing two APIs: 1. request_release(): tells ManagedBuffer if I have the last reference, release the buffer now. 2. release():

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: In order to have a basis for discussion, I've set up a repo at http://hg.python.org/features/pep-3118#memoryview with an implementation of PyManagedBuffer. The whole test suite passes, also with refleak counting and Valgrind.

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Stefan Krah
Changes by Stefan Krah stefan-use...@bytereef.org: Added file: http://bugs.python.org/file22564/bbe70ca4e0e5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread STINNER Victor
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10181 ___ ___

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick Coghlan rep...@bugs.python.org wrote: The reason redirecting all requests to the underlying object doesn't work is because repeated calls to getbuffer on mutable objects are allowed to return *different* answers. Assume we have a

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Pauli Virtanen
Pauli Virtanen p...@iki.fi added the comment: @skrah: Yes, Numpy exposes only a single buffer per object. Making this a requirement in the PEP would probably be a sane change, as there is probably little real-world need to allow it behave otherwise. Comment on the patch: it seems you do not

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Pauli Virtanen rep...@bugs.python.org wrote: Comment on the patch: it seems you do not track the re-export count in memory_getbuf: a = memoryview(obj) b = numpy.asarray(a) a.release() b[0] = 123 # -- BOOM: the

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: It might be worth postponing the multi-dimensional support to a second patch, though. If we can get the buffer lifecycle solid first then that provides a better foundation for any further development. --

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: As far as the rule of disallowing shape changes while a buffer is exported, I actually think that's a more sane approach as well. However, I've been burned enough times by going nobody would be insane enough to rely on that, would they? that

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: [I agree that multi-dimensional support should not be part of this patch. I was thinking about creating a separate branch for that.] Nick Coghlan rep...@bugs.python.org wrote: As far as the rule of disallowing shape changes while a

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Nice work with the patch Stefan - I've added a few review comments (including a suggestion on how to deal with the GetContiguous problem). One idea that review did prompt is that if we aren't going back to the original object for fresh buffer

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: Is there anything stopping us just storing the flags on PyManagedBuffer? It's OK if the construction API requires the flag information in addition to the Py_buffer struct. -- ___ Python tracker

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Pauli Virtanen
Pauli Virtanen p...@iki.fi added the comment: @skrah: Ahh, this always happens when I don't run it :) But my point stands -- the reason why it does not crash with Numpy is that Numpy calls PyMemoryView_FromObject to obtain a new memoryview and then uses PyMemoryView_GET_BUFFER. Along this

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-07-04 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: On Tue, Jul 5, 2011 at 12:36 AM, Pauli Virtanen rep...@bugs.python.org wrote: Slicing memoryviews can invalidate the contiguity flags, and no-strides flags, so some checks are still probably needed in `memory_getbuf`. That makes sense, so

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick, you know a lot about this issue and I'm probably missing many things here. I misunderstood your concept of PyManagedBuffer, so my previous posting might have been hard to understand. I'd appreciate if you (or anyone in this thread)

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Pauli Virtanen
Pauli Virtanen p...@iki.fi added the comment: skrah writes: I think slicing (esp. multidimensional slicing) would be greatly simplified if we added a requirement for the *exporting* object to provide a sliced view. (The same applies to sub-views, also see source comments below [1]). For

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: I'll try to do a summary of the conversation so far, since it's quite long and hard to follow. The basic issue is that memoryview needs to support copying and slicing that creates a new memoryview object. The major problem with that is that

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Pauli Virtanen rep...@bugs.python.org wrote: skrah writes: For example, an exporting object could provide a sliced view by adding a getslicedbufferproc to PyBufferProcs: int PyObject_GetSlicedBuffer(PyObject *obj, Py_buffer

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: Nick Coghlan rep...@bugs.python.org wrote: [Snip liberally] The lifecycle problem is unrelated to the details of the buffer *contents* though - it's entirely about the fact that clients can't safely copy all those pointers (as some

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Le Mon, 27 Jun 2011 13:17:57 +, Nick Coghlan rep...@bugs.python.org a écrit : The TL;DR version of the above is that I would like to see it become: typedef struct { PyObject_HEAD PyManagedBuffer source_data; // shared read-only

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Lenard Lindstrom
Lenard Lindstrom le...@telus.net added the comment: Using Python reference counting and the garbage collector to control when PyBuffer_Release is called has problems. First, it assumes the CPython interpreter will always use reference counting. Second, PyBuffer_Release may never get called if

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Pauli Virtanen
Pauli Virtanen p...@iki.fi added the comment: Lenard Lindstrom writes: Using Python reference counting and the garbage collector to control when PyBuffer_Release is called has problems. That's not what's being suggested. The refcounting discussed here is an implementation detail internal to

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Lenard Lindstrom
Lenard Lindstrom le...@telus.net added the comment: It would if the proposed PyManagedBuffer only releases the Py_buffer struct - calls PyBuffer_Release - when its Python reference count goes to zero. So a separate reference count will be maintained instead. --

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-27 Thread Nick Coghlan
Nick Coghlan ncogh...@gmail.com added the comment: memoryview.release() will raise an exception if you call it when the underlying PyManagedBuffer instance has a refcount 1. So the explicit memory management still works, and if you mess it up by leaving dangling references around you'll run

[issue10181] Problems with Py_buffer management in memoryobject.c (and elsewhere?)

2011-06-26 Thread Stefan Krah
Stefan Krah stefan-use...@bytereef.org added the comment: [Nick] Another idea we may want to revisit is the PyManagedBuffer concept, which would be a true PyObject that existed solely to simplify sharing of Py_buffer structs. If memoryview used such an object internally, then copying and

  1   2   >