Re: [Python-Dev] O(1) deletes from the front of bytearray (was: Re: Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor)
On 13.10.16 00:14, Nathaniel Smith wrote: AFAIK basically the only project that would be affected by this is PyPy, And MicroPython. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] O(1) deletes from the front of bytearray (was: Re: Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor)
> AFAIK basically the only project that would be affected by this is > PyPy, and I when I asked on #pypy they said: > > njs`: I think we either plan to or already support this > > so I'm not sure why this is controversial. FYI, I had filed the feature request on PyPy issue tracker. https://bitbucket.org/pypy/pypy/issues/2367/del-bytearray-n-should-be-optimized In case of micropython, it doesn't support deletion for now. https://github.com/micropython/micropython/blob/b9672bcbe8dd29b61326af8eb026df4d10a8f0ce/py/objarray.c#L371-L378 -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Playing games with reference counts (was Re: PyWeakref_GetObject() borrows its reference from... whom?)
On 10/10/2016 10:38 PM, Chris Angelico wrote: On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings wrote: These hacks where we play games with the reference count are mostly removed in my branch. That's exactly what I would have said, because I was assuming that refcounts would be accurate. I'm not sure what you mean by "play games with", By "playing games with reference counts", I mean code that purposely doesn't follow the rules of reference counting. Sadly, there are special cases that apparently *are* special enough to break the rules. Which made implementing "buffered reference counting" that much harder. I currently know of two examples of this in CPython. In both instances, an object has a reference to another object, but *deliberately* does not increase the reference count of the object, in order to prevent keeping the other object alive. The implementation relies on the GIL to preserve correctness; without a GIL, it was much harder to ensure this code was correct. (And I'm still not 100% I've done it. More thinking needed.) Those two examples are: 1. PyWeakReference objects. The wr_object pointer--the "reference" held by the weak reference object--points to an object, but does not increment the reference count. Worse yet, as already observed, PyWeakref_GetObject() and PyWeakref_GET_OBJECT() don't increment the reference count, an inconvenient API decision from my perspective. 2. "Interned mortal" strings. When a string is both interned *and* mortal, it's stored in the static "interned" dict in unicodeobject.c--as both key and value--and then its's DECREF'd twice so those two references don't count. When the string is destroyed, unicode_dealloc resurrects the string, reinstating those two references, then removes it from the "interned" dict, then destroys the string as normal. To support these, I've implemented what is effectively a secondary, atomic-only reference count. It seems to work. (And yes that means all objects are now 8 bytes bigger. Let me worry about memory consumption later, m'kay?) Resurrecting object also gave me a headache in the Gilectomy with this buffered reference counting scheme, but I think I have that figured out too. When you resurrect an object, it's generally because you're going to expose it to other subsystems that may incr / decr / otherwise inspect the reference count. Which means that code may buffer reference count changes. Which means you can't immediately destroy the object anymore. So: when you resurrect, you set the new reference count, you also set a flag saying "I've already been resurrected", you pass it in to that other code, you then drop your references with Py_DECREF, and you exit. Your dealloc function will get called again later; you then see you've already done that first resurrection, and you destroy as normal. Curiously enough, the typeobject actually needs to do this twice: once for tp_finalize, once for tp_del. (Assuming I didn't completely misunderstand what the code was doing.) My struggles continue, //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] O(1) deletes from the front of bytearray (was: Re: Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor)
On Wed, 12 Oct 2016 14:14:48 -0700 Nathaniel Smith wrote: > > The proposal is that it should be documented as being part of the > language spec starting in 3.4 (or whatever). So applications that > support Python 2.7 can't rely on it, sure. But if I have an > application that requires, say, 3.5+ but I don't want to depend on > CPython-only implementation details, then I'm still allowed to use it. > > AFAIK basically the only project that would be affected by this is > PyPy, and I when I asked on #pypy they said: > > njs`: I think we either plan to or already support this > > so I'm not sure why this is controversial. The main reason it could be controversial is if someone finds out another optimization that is made difficult or impossible by the O(1) delete-at-front guarantee. That sounds unlikely, though, since the general structure of a bytearray is constrained by other factors as well (such as the C API and the buffer API). By the way, to answer another question, the reason this wasn't made part of the spec originally is that the feature in itself was already contentious (see issue tracker discussion). Now that more people seem to get interested in network programming, they seem to understand the point ;-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging Python scripts with GDB on OSX
Hi Alexandru, I stumbled over this question a little late by chance. There is the possibility to use GDB, but it is most likely that you want to use python's pdb module, instead. Only in rare cases, when debugging the interpreter itself, you use gdb. For debugging Python code, use pdb or something better. Sent from my Ei4Steve > On Jul 6, 2016, at 18:14, Alexandru Croitor wrote: > > Hello, > > I'm interested to find out if debugging Python scripts with GDB is supported > on OSX at all? > > I'm referring to the functionality described on > https://wiki.python.org/moin/DebuggingWithGdb and on > http://fedoraproject.org/wiki/Features/EasierPythonDebugging. > > I've tried so far various combinations of pre-compiled GDB from the homebrew > package manager, locally-compiled GDB from homebrew, as well as locally > compiled GDB from MacPorts, together with a pre-compiled Python 2.7, > homebrew-compiled 2.7, and custom compiled Python 2.7 from the official > source tarball. > > My results so far were not successful. The legacy GDB commands to show a > python stack trace or the local variables - do not work. And the new GDB > commands (referenced on the Fedora project page) are not present at all in > any of the GDB versions. > > I've checked the python CI build bot tests, and it seems the new GDB commands > are only successfully tested on Linux machines, and are skipped on FreeBSD, > OS X, and Solaris machines. > > Are the new python <-> GDB commands specific to Linux? > Are there any considerations to take in regards to debug symbols for Python / > GDB on OSX? > > Has anyone attempted what I'm trying to do? > > I would be grateful for any advice. > > And I apologize if my choice of the mailing lists is not the best. > > Regards, Alex. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Update to PEP 1 re: content type
Hello, PEP 1 states that plain/text is an acceptable value for a PEP's content type, and it is the default value if no content type is specified. May I propose adding something along the line of: "new PEPs should use restructured Text format", and that reST format becomes the default. Based on couple tickets in peps repo, it seems like the reST format is preferred anyway, and there have been thoughts about converting text based PEPs into reST format https://github.com/python/peps/issues/4 https://github.com/python/peps/issues/2 I hope this is not too controversial. Discussed this with Guido earlier. He is supportive, and asked me to send email to python-dev :) Thanks Mariatta Wijaya ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Playing games with reference counts (was Re: PyWeakref_GetObject() borrows its reference from... whom?)
On Thu, Oct 13, 2016 at 4:43 AM Larry Hastings wrote: > > On 10/10/2016 10:38 PM, Chris Angelico wrote: > > On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings > wrote: > > These hacks where we play games with the > reference count are mostly removed in my branch. > > That's exactly what I would have said, because I was assuming that > refcounts would be accurate. I'm not sure what you mean by "play games > with", > > > By "playing games with reference counts", I mean code that purposely > doesn't follow the rules of reference counting. Sadly, there are special > cases that apparently *are* special enough to break the rules. Which made > implementing "buffered reference counting" that much harder. > > I currently know of two examples of this in CPython. In both instances, > an object has a reference to another object, but *deliberately* does not > increase the reference count of the object, in order to prevent keeping the > other object alive. The implementation relies on the GIL to preserve > correctness; without a GIL, it was much harder to ensure this code was > correct. (And I'm still not 100% I've done it. More thinking needed.) > > Those two examples are: > >1. PyWeakReference objects. The wr_object pointer--the "reference" >held by the weak reference object--points to an object, but does not >increment the reference count. Worse yet, as already observed, >PyWeakref_GetObject() and PyWeakref_GET_OBJECT() don't increment the >reference count, an inconvenient API decision from my perspective. > > That a PyWeakReference object does not increment the reference count is the entire point of a weakref. The object wouldn't be destroyed and break the weak reference otherwise. Weak references could be implemented in a different manner - coordinate with the garbage collector to consider things who's only references come from weakrefs as collectable. That'd be an internal overhaul of the weakref implementation and potentially the gc. > >1. >2. "Interned mortal" strings. When a string is both interned *and* >mortal, it's stored in the static "interned" dict in unicodeobject.c--as >both key and value--and then its's DECREF'd twice so those two references >don't count. When the string is destroyed, unicode_dealloc resurrects the >string, reinstating those two references, then removes it from the >"interned" dict, then destroys the string as normal. > > yow. i don't even want to know the history of that one... Resurrecting object also gave me a headache in the Gilectomy with this > buffered reference counting scheme, but I think I have that figured out > too. When you resurrect an object, it's generally because you're going to > expose it to other subsystems that may incr / decr / otherwise inspect the > reference count. Which means that code may buffer reference count > changes. Which means you can't immediately destroy the object anymore. > So: when you resurrect, you set the new reference count, you also set a > flag saying "I've already been resurrected", you pass it in to that other > code, you then drop your references with Py_DECREF, and you exit. Your > dealloc function will get called again later; you then see you've already > done that first resurrection, and you destroy as normal. Curiously enough, > the typeobject actually needs to do this twice: once for tp_finalize, once > for tp_del. (Assuming I didn't completely misunderstand what the code was > doing.) > > kudos for trying to understand this. resurrection during destruction or finalization hurts my brain even though in many ways it makes sense. -gps ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On Fri, Oct 14, 2016 at 5:09 AM, Mariatta Wijaya wrote: > PEP 1 states that plain/text is an acceptable value for a PEP's content > type, and it is the default value if no content type is specified. > > May I propose adding something along the line of: "new PEPs should use > restructured Text format", and that reST format becomes the default. +1. It's already the de facto standard, and I support promoting that to de jure. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
Confirming, +1 from me. Barry, Nick? You two are the most active authors of PEP 1. On Thu, Oct 13, 2016 at 11:43 AM, Chris Angelico wrote: > On Fri, Oct 14, 2016 at 5:09 AM, Mariatta Wijaya > wrote: >> PEP 1 states that plain/text is an acceptable value for a PEP's content >> type, and it is the default value if no content type is specified. >> >> May I propose adding something along the line of: "new PEPs should use >> restructured Text format", and that reST format becomes the default. > > +1. It's already the de facto standard, and I support promoting that to de > jure. > > ChrisA > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On Oct 13, 2016, at 01:32 PM, Guido van Rossum wrote: >Confirming, +1 from me. Barry, Nick? You two are the most active >authors of PEP 1. +1 for text/x-rst being the preferred. We'd need some time to make it the default, but I'm generally in favor of that happening. Is anybody actively working on issue #4? https://github.com/python/peps/issues/4 Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?
On Mon, Oct 10, 2016, at 14:04, MRAB wrote: > Instead of locking the object, could we keep the GIL, but have it > normally released? > A thread could then still call a function such as PyWeakref_GetObject() > that returns a borrowed reference, but only if it's holding the GIL. It > would be able to INCREF the reference before releasing the GIL again. So you need to get/release the GIL just to run a slightly faster function that doesn't bother with an extra incref/defcref pair? I think anyone willing to make those changes would be willing to switch to a non-borrowing version of that same function, and do an explicit DECREF if that is really what they wanted. On Tue, Oct 11, 2016 at 5:24 AM, Random832 wrote: > So, what stops the other thread which never asks for the GIL from > blowing away the reference? Or is this a special kind of lock that you > can "assert isn't locked" without locking it for yourself, and > INCREF/DECREF does so? On Mon Oct 10 15:36:59 EDT 2016, Chris Angelico wrote: > "assert isn't locked" is pretty cheap Yeah, but so is INCREF/DECREF on memory that is almost certainly in cache anyhow, because you're using the object right next to it. The write part hurts, particularly when trying to use multiple cores with shared memory, but any sort of indirection (even separating the refcount from the object, to allow per-core counters) ... well, it doesn't take much at all to be worse than INCREF/DECREF in even the normal case, let alone amortized across the the "drat, this object now has to be handled specially" cases. Imagine two memory pools, one for "immortal" objects (such as None) that won't be collected, and so don't need their memory dirtied when you INCREF/DECREF. Alas, now *every* INCREF and DECREF has to branch on the address to tell whether or not it should be a no-op. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On 10/13/2016 4:38 PM, Barry Warsaw wrote: On Oct 13, 2016, at 01:32 PM, Guido van Rossum wrote: Confirming, +1 from me. Barry, Nick? You two are the most active authors of PEP 1. +1 for text/x-rst being the preferred. We'd need some time to make it the default, but I'm generally in favor of that happening. The major issue is making knowledge of rst and sphinx a requirement for submitting a PEP. I suggest making sure that PEP 1 exemplifies the preferred usage of rst in PEPs and then suggesting using it as model. A related issue is plain text for posting. Initial pre-drafts can be posted to python-ideas before being marked up. Other than copying the browser display of the .html conversion (which requires a bit of cleanup after pasting), what is the easiest way to get post-markup plain text? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On Oct 13, 2016, at 05:31 PM, Terry Reedy wrote: >The major issue is making knowledge of rst and sphinx a requirement for >submitting a PEP. I suggest making sure that PEP 1 exemplifies the preferred >usage of rst in PEPs and then suggesting using it as model. https://github.com/python/peps/pull/115 >A related issue is plain text for posting. Initial pre-drafts can be posted >to python-ideas before being marked up. Other than copying the browser >display of the .html conversion (which requires a bit of cleanup after >pasting), what is the easiest way to get post-markup plain text? I don't understand the question: isn't the whole point of reST that it's a easily readable markup language? Just post the reST! Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On Thu, Oct 13, 2016 at 3:59 PM, Barry Warsaw wrote: > I don't understand the question: isn't the whole point of reST that it's a > easily readable markup language? Just post the reST! +1 -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] O(1) deletes from the front of bytearray (was: Re: Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas] Adding bytes.frombuffer() constructor)
On 13/10/2016 11:41, Serhiy Storchaka wrote: On 13.10.16 00:14, Nathaniel Smith wrote: AFAIK basically the only project that would be affected by this is PyPy, And MicroPython. And Jython, except that from the start its implementation of bytearray deferred resizing until the proportion unused space reaches some limit. I think that should make it O(log N) on average to delete (or add) a byte, at either end of a buffer of size N,. However, observations with timeit() look constant up to the point I run out of heap. Jeff Allen ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] single dispatch for instance methods
Hi All, It would be nice if the @singledispatch decorator worked on instance methods. I have a reference implementation on pypi that does this ( https://pypi.python.org/pypi/methoddispatch). I posted this message to python ideas a month ago ( https://mail.python.org/pipermail/python-ideas/2016-September/042466.html) and got 100% support from 1 response. Not sure if I should proceed with writing a PEP. I am happy to do all the work if I know it will be accepted. Cheers Tim ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] single dispatch for instance methods
Just one reply seems pretty weak of a push. However, you lose nothing by submitting it on the issue tracker: https://bugs.python.org I don’t have a use case for this myself, but we’ll see :) -Emanuel From: Python-Dev [mailto:python-dev-bounces+vgr255=live...@python.org] On Behalf Of Tim Mitchell Sent: Thursday, October 13, 2016 9:11 PM To: python-dev@python.org Subject: [Python-Dev] single dispatch for instance methods Hi All, It would be nice if the @singledispatch decorator worked on instance methods. I have a reference implementation on pypi that does this (https://pypi.python.org/pypi/methoddispatch). I posted this message to python ideas a month ago (https://mail.python.org/pipermail/python-ideas/2016-September/042466.html) and got 100% support from 1 response. Not sure if I should proceed with writing a PEP. I am happy to do all the work if I know it will be accepted. Cheers Tim ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to PEP 1 re: content type
On 10/13/2016 5:59 PM, Barry Warsaw wrote: On Oct 13, 2016, at 05:31 PM, Terry Reedy wrote: The major issue is making knowledge of rst and sphinx a requirement for submitting a PEP. I suggest making sure that PEP 1 exemplifies the preferred usage of rst in PEPs and then suggesting using it as model. https://github.com/python/peps/pull/115 A related issue is plain text for posting. Initial pre-drafts can be posted to python-ideas before being marked up. Other than copying the browser display of the .html conversion (which requires a bit of cleanup after pasting), what is the easiest way to get post-markup plain text? I don't understand the question: isn't the whole point of reST that it's a easily readable markup language? Just post the reST! OK with me if that becomes the norm. I don't think it is now. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
