[Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
Hi, all. Currently, deque and defaultdict have only C implementation. Python implementations should provide _collections.deque and _collections.defaultdict. Like that, how about removing OrderedDict Pure Python implementation from stdlib and require it to implementation? ## Pros ### Thread safety AFAIK, there are no thread safety guarantee in OrderedDict. I don't look carefully, but some methods seems thread unsafe. I'm considering adding `OrderedDict.lru_get(key, default=None)` which atomically lookup key and move the key to end if found. It can be used when functools.lru_cahce can't be used. (see http://bugs.python.org/issue28193 ) And thread safety is very important for such method. Anyway, thread safety will improve usability of OrderedDict significantly, like defaultdict. ### Less maintenance cost of test_ordered_dict. I'm sending pull request for removing doubly linked list from C OrderedDict, for faster creatin, iteration, and reduce memory usage by 1/2. See https://bugs.python.org/issue31265 While implementing it, I noticed that current test_ordered_dict has some tests about implementation detail without @cpython_only decorator. For example: https://github.com/python/cpython/blob/fcd97d44382df520e39de477a5ec99dd89c3fda8/Lib/test/test_ordered_dict.py#L522-L530 While current test expects KeyError, my pull request (and PyPy's OrderedDict) doesn't raise KeyError because inconsistency between doubly linked list and dict never happens. PyPy changed the test: https://bitbucket.org/pypy/pypy/src/ac3e33369ba0aa14278a6a3e8f2add09590d358c/lib-python/3/test/test_ordered_dict.py?at=py3.6&fileviewer=file-view-default#test_ordered_dict.py-525:542 My pull request has same change: https://github.com/methane/cpython/pull/9/files#diff-23c28e7fa52967682669d9059dc977ed Maintain compatibility with odd behavior of Pure Python implementation is not constructive job not only for us, but also other Python implementations. ### `import collections` bit faster. Pure Python implementation has 5 classes (OrderedDict, _Link, _OrderedDictKeyViews, _OrderedDictItemsView, and _OrderedDictValuesView). Three of them inheriting from ABC. So it makes importing collections bit slower. ## Cons * All Python 3.7 implementations should provide _collections.OrderedDict PyPy has it already. But I don't know about micropython. Regards, INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
05.09.17 11:38, INADA Naoki пише: ## Cons * All Python 3.7 implementations should provide _collections.OrderedDict PyPy has it already. But I don't know about micropython. Current C implementation of OrderedDict is not safe regarding using mutating dict methods (or dict C API) like dict.__setitem__ or PyDict_SetItem. Using them can cause hangs or segfaults. See issue24726 and issue25410. I hope your implementation will solve these issues, but there may be others. While the C implementation still is not enough mature, we should allow users that encountered one of such issues to use pure Python implementation which is free from hangs and segfaults. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
I thought the decision a few years ago was that all modules that have a C library for performance reasons should also have a Python version? Did this decision change at some point? (just curious). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] To reduce Python "application" startup time
Hi, While I can't attend to sprint, I saw etherpad and I found Neil Schemenauer and Eric Snow will work on startup time. I want to share my current knowledge about startup time. For bare (e.g. `python -c pass`) startup time, I'm waiting C implementation of ABC. But application startup time is more important. And we can improve them with optimize importing common stdlib. Current `python -v` is not useful to optimize import. So I use this patch to profile import time. https://gist.github.com/methane/e688bb31a23bcc437defcea4b815b1eb With this profile, I tried optimize `python -c 'import asyncio'`, logging and http.client. https://gist.github.com/methane/1ab97181e74a33592314c7619bf34233#file-0-optimize-import-patch With this small patch: logging: 14.9ms -> 12.9ms asyncio: 62.1ms -> 58.2ms http.client: 43.8ms -> 36.1ms I haven't created pull request yet. (Can I create without issue, as trivial patch?) I'm very busy these days, maybe until December. But I hope this report helps people working on optimizing startup time. Regards, INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, Sep 5, 2017 at 9:58 PM, Simon Cross wrote: > I thought the decision a few years ago was that all modules that have a C > library for performance reasons should also have a Python version? Did this > decision change at some point? (just curious). But in this case, not only for optimize, but also better behavior. Pure Python version is not thread safe even for `od[k] = v`. It's very difficult to write thread safe complex container in Pure Python. Regards, INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, Sep 5, 2017 at 8:48 PM, Serhiy Storchaka wrote: > 05.09.17 11:38, INADA Naoki пише: >> >> ## Cons >> >> * All Python 3.7 implementations should provide _collections.OrderedDict >>PyPy has it already. But I don't know about micropython. > > > Current C implementation of OrderedDict is not safe regarding using mutating > dict methods (or dict C API) like dict.__setitem__ or PyDict_SetItem. Using > them can cause hangs or segfaults. See issue24726 and issue25410. I hope > your implementation will solve these issues, but there may be others. Thanks for sharing these issues. My pull request is still early stage and may have some bugs. But I think it will solve many of issues because most odd behaviors comes from difficult of consistency between linked list and dict. > While > the C implementation still is not enough mature, we should allow users that > encountered one of such issues to use pure Python implementation which is > free from hangs and segfaults. > I see. Maybe, we can move Pure Python OrderedDict to collections._odict module, and collections/__init__.py can be like this: from _collections import OrderedDict # Pure Python implementation can be used when C implementation cause segfault. # from collections._odict import OrderedDict > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, 5 Sep 2017 17:38:06 +0900 INADA Naoki wrote: > > Like that, how about removing OrderedDict Pure Python implementation > from stdlib and require it to implementation? I don't like this. The C version of OrderedDict is probably very hard to read, while the Python version helps understand the semantics. Also, when devising a new API, it's much easier to first try it out on the Python version. > I'm considering adding `OrderedDict.lru_get(key, default=None)` which > atomically lookup key and move the key to end if found. The Python version needn't have the same atomicity requirements as the C version. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, Sep 5, 2017 at 5:58 AM, Simon Cross wrote: > I thought the decision a few years ago was that all modules that have a C > library for performance reasons should also have a Python version? Did this > decision change at some point? (just curious). > It was never meant as a hard rule. IIRC the actual rule is more that *if* you have both a C and a Python version they need to supply the same API, *and* the naming should be e.g. _pickle (C) and pickle (Python), *and* the Python version should automatically replace itself with the C version when the latter exists (e.g. by putting an import * at the end with a try/except around it). There are tons of modules that only have a C version and there's no need to change this -- and I'm fine with occasionally a module removing the Python version when the C version is deemed sufficiently stable and compatible. (In some cases a move in the opposite direction may also be reasonable. No two cases are exactly alike.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
On 5 September 2017 at 15:02, INADA Naoki wrote: > Hi, > > [...] > > For bare (e.g. `python -c pass`) startup time, I'm waiting C > implementation of ABC. > Hi, I am not sure I will be able to finish it this week, also this depends on fixing interactions with ABC caches in ``typing`` first (as I mentioned on b.p.o., currently ``typing`` "aggressively" uses private ABC API). -- Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Compiling without multithreading support -- still useful?
Hello, It's 2017 and we are still allowing people to compile CPython without threads support. It adds some complication in several places (including delicate parts of our internal C code) without a clear benefit. Do people still need this? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling without multithreading support -- still useful?
I proposed to drop the --without-threads option multiple times. I worked on tiny and cheap embedded devices and we used Python *with* threads for concurrency. Many Python features require threads, like asyncio and multiprocessing. Also subprocess.communicate() on Windows, no? I'm strongly in favor of dropping this option from Python 3.7. It would remove a lot of code! Victor 2017-09-05 18:36 GMT+02:00 Antoine Pitrou : > > Hello, > > It's 2017 and we are still allowing people to compile CPython without > threads support. It adds some complication in several places > (including delicate parts of our internal C code) without a clear > benefit. Do people still need this? > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiling without multithreading support -- still useful?
On 2017-09-05 09:42, Victor Stinner wrote: > I proposed to drop the --without-threads option multiple times. I > worked on tiny and cheap embedded devices and we used Python *with* > threads for concurrency. Many Python features require threads, like > asyncio and multiprocessing. Also subprocess.communicate() on Windows, > no? > > I'm strongly in favor of dropping this option from Python 3.7. It > would remove a lot of code! +1 These days, tiny embedded devices can make use of MicroPython. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, Sep 5, 2017 at 1:38 AM, INADA Naoki wrote: > Like that, how about removing OrderedDict Pure Python implementation > from stdlib and require it to implementation? -1 Like Antoine, I consider the pure Python implementation to be valuable. Furthermore, the pure Python implementation is the reference, so its behavior is idiomatic. > ### Thread safety > > AFAIK, there are no thread safety guarantee in OrderedDict. > I don't look carefully, but some methods seems thread unsafe. What isn't thread-safe? I know that Raymond has a good understanding of this area. For instance, he was very clear about re-entrancy concerns when I was working on the C implementation. I recommend getting feedback from him on this. FWIW, I don't recall any bugs related to thread-safety in OrderedDict, even though it's been around a while. > [snip] > > ### Less maintenance cost of test_ordered_dict. > > [snip] I don't find this to be a strong argument. If there are concerns with the reference behavior then those should be addressed rather than used to justify removing the implementation. > ### `import collections` bit faster. > > [snip] This is not a strong argument. The difference is not significant enough to warrant removing the reference implementation. So, again, I'm against removing the pure Python implementation of OrderedDict. I highly recommend making sure you have Raymond's cooperation before making changes in the collections module. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Tue, Sep 5, 2017 at 12:13 PM, Eric Snow wrote: > On Tue, Sep 5, 2017 at 1:38 AM, INADA Naoki wrote: >> Like that, how about removing OrderedDict Pure Python implementation >> from stdlib and require it to implementation? > > -1 > > Like Antoine, I consider the pure Python implementation to be > valuable. Furthermore, the pure Python implementation is the > reference, so its behavior is idiomatic. -1 as well, I'm in favor of keeping Python implementations of things implemented in C, no matter the order in which they are added. >> ### Less maintenance cost of test_ordered_dict. >> >> [snip] > > I don't find this to be a strong argument. If there are concerns with > the reference behavior then those should be addressed rather than used > to justify removing the implementation. I agree here as well. We have plenty of things implemented in both Python and C, and have helpers in test.support to make it (relatively, at least) easy to test both. PEP 399 is also relevant here, I think. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
On 9/5/2017 9:02 AM, INADA Naoki wrote: But application startup time is more important. And we can improve them with optimize importing common stdlib. Current `python -v` is not useful to optimize import. So I use this patch to profile import time. https://gist.github.com/methane/e688bb31a23bcc437defcea4b815b1eb With this profile, I tried optimize `python -c 'import asyncio'`, logging and http.client. https://gist.github.com/methane/1ab97181e74a33592314c7619bf34233#file-0-optimize-import-patch With this small patch: logging: 14.9ms -> 12.9ms asyncio: 62.1ms -> 58.2ms http.client: 43.8ms -> 36.1ms I haven't created pull request yet. (Can I create without issue, as trivial patch?) Trivial, no-issue PRs are meant for things like typo fixes that need no discussion or record. Moving imports in violation of the PEP 8 rule, "Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants", is not trivial. Doing so voluntarily for speed, as opposed to doing so necessarily to avoid circular import errors, is controversial. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On 9/5/2017 10:40 AM, Guido van Rossum wrote: On Tue, Sep 5, 2017 at 5:58 AM, Simon Cross mailto:hodgestar+python...@gmail.com>> wrote: I thought the decision a few years ago was that all modules that have a C library for performance reasons should also have a Python version? Did this decision change at some point? (just curious). It was never meant as a hard rule. IIRC the actual rule is more that *if* you have both a C and a Python version they need to supply the same API, *and* the naming should be e.g. _pickle (C) and pickle (Python), *and* the Python version should automatically replace itself with the C version when the latter exists (e.g. by putting an import * at the end with a try/except around it). *And* both versions should be tested in the test file ;-). -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
2017-09-05 6:02 GMT-07:00 INADA Naoki : > Hi, > > While I can't attend to sprint, I saw etherpad and I found > Neil Schemenauer and Eric Snow will work on startup time. > > I want to share my current knowledge about startup time. > > For bare (e.g. `python -c pass`) startup time, I'm waiting C > implementation of ABC. > > But application startup time is more important. And we can improve > them with optimize importing common stdlib. > > Current `python -v` is not useful to optimize import. > So I use this patch to profile import time. > https://gist.github.com/methane/e688bb31a23bcc437defcea4b815b1eb > > With this profile, I tried optimize `python -c 'import asyncio'`, logging > and http.client. > > https://gist.github.com/methane/1ab97181e74a33592314c7619bf342 > 33#file-0-optimize-import-patch > > This patch moves a few imports inside functions. I wonder whether that kind of change actually helps with real applications—doesn't any real application end up importing the socket module anyway at some point? > With this small patch: > > logging: 14.9ms -> 12.9ms > asyncio: 62.1ms -> 58.2ms > http.client: 43.8ms -> 36.1ms > > I haven't created pull request yet. > (Can I create without issue, as trivial patch?) > > I'm very busy these days, maybe until December. > But I hope this report helps people working on optimizing startup time. > > Regards, > > INADA Naoki > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jelle.zijlstra%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
First of all, I saw enough -1 so I gave up about removing.
Following reply is just a technical topic.
On Wed, Sep 6, 2017 at 2:13 AM, Eric Snow wrote:
[snip]
>
> Like Antoine, I consider the pure Python implementation to be
> valuable. Furthermore, the pure Python implementation is the
> reference, so its behavior is idiomatic.
>
What is *idiomatic*? For example, in this test case:
https://github.com/python/cpython/blob/564a2c68add64ebf2e558a54f5697513b19293cb/Lib/test/test_ordered_dict.py#L575-L582
def test_dict_delitem(self):
OrderedDict = self.OrderedDict
od = OrderedDict()
od['spam'] = 1
od['ham'] = 2
dict.__delitem__(od, 'spam')
with self.assertRaises(KeyError):
repr(od)
Since dict.__delitem__ is same to OrderedDict.__delitem__ in PyPy
implementation,
repr(od) doesn't raise KeyError.
import collections
od = collections.OrderedDict()
od['spam'] = 1
od['ham'] = 2
dict.__delitem__(od, 'spam')
repr(od)
"OrderedDict([('ham', 2)])"
Do you think this behavior is not idiomatic?
I feel both are OK and there are no idiomatic behavior.
The tested behavior seems just an implementation detail.
At least, PyPy modified this test as I wrote in previous mail.
It makes sense to me.
>> ### Thread safety
>>
>> AFAIK, there are no thread safety guarantee in OrderedDict.
>> I don't look carefully, but some methods seems thread unsafe.
>
> What isn't thread-safe? I know that Raymond has a good understanding
> of this area. For instance, he was very clear about re-entrancy
> concerns when I was working on the C implementation. I recommend
> getting feedback from him on this. FWIW, I don't recall any bugs
> related to thread-safety in OrderedDict, even though it's been around
> a while.
For example, when one thread do `od[k1] = v1` and another thread do
`od[k2] = v2`,
result should be equivalent to one of `od[k1] = v1; od[k2] = v2;` or
`od[k2] = v2; od[k1] = v1`. And internal data structure should be consistent.
But see current Pure Python implementation:
def __setitem__(self, key, value,
dict_setitem=dict.__setitem__, proxy=_proxy, Link=_Link):
'od.__setitem__(i, y) <==> od[i]=y'
# Setting a new item creates a new link at the end of the linked list,
# and the inherited dictionary is updated with the new key/value pair.
if key not in self:
self.__map[key] = link = Link()
root = self.__root
last = root.prev
link.prev, link.next, link.key = last, root, key
last.next = link
root.prev = proxy(link)
dict_setitem(self, key, value)
If thread is switched right after `root = self.__root`, either k1 or k2 is
disappeared in linked list.
More worth, if thread switch is happen right after `last = root.prev`,
linked list will be broken.
Fixing such issue is not easy job. See
http://bugs.python.org/issue14976 for example.
On the other hand, thanks to GIL, C implementation (not only mine, but
also your's)
doesn't have such issue.
Sometime, writing reentrant and threadsafe container in C is easier than
in Pure Python.
Regards,
P.S.
Since it's over 3 a.m. in Japan, I'll read and reply other mails tomorrow.
Sorry for delay.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Version and Last-Modified headers are no longer required in PEPs.
The Version and Last-Modified headers required by PEP1 used to be maintained by the version control system, but this is not true now that we've switched to git. We are therefore deprecating these headers and have removed them from PEP1. The PEP generation script now considers them to be optional. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On Wed, 6 Sep 2017 03:20:43 +0900 INADA Naoki wrote: > What is *idiomatic*? For example, in this test case: > > https://github.com/python/cpython/blob/564a2c68add64ebf2e558a54f5697513b19293cb/Lib/test/test_ordered_dict.py#L575-L582 > > def test_dict_delitem(self): > OrderedDict = self.OrderedDict > od = OrderedDict() > od['spam'] = 1 > od['ham'] = 2 > dict.__delitem__(od, 'spam') > with self.assertRaises(KeyError): > repr(od) > > Since dict.__delitem__ is same to OrderedDict.__delitem__ in PyPy > implementation, > repr(od) doesn't raise KeyError. Since this test is testing an implementation detail, it can safely be moved to a test class specific to the pure Python implementation. > For example, when one thread do `od[k1] = v1` and another thread do > `od[k2] = v2`, > result should be equivalent to one of `od[k1] = v1; od[k2] = v2;` or > `od[k2] = v2; od[k1] = v1`. And internal data structure should be consistent. I agree the pure Python OrderedDict is not thread-safe. But who said it should? And, actually, are you sure the C implementation is? GIL switches can happen at many points. A simple Py_DECREF, or perhaps even a tuple allocation can trigger a GC run that may release the GIL at some point in a finalizer function. > Sometime, writing reentrant and threadsafe container in C is easier than > in Pure Python. The C and Python versions needn't make exactly the same guarantees. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
On Tue, Sep 5, 2017 at 11:13 AM, Jelle Zijlstra wrote: > > > 2017-09-05 6:02 GMT-07:00 INADA Naoki : >> With this profile, I tried optimize `python -c 'import asyncio'`, logging >> and http.client. >> >> >> https://gist.github.com/methane/1ab97181e74a33592314c7619bf34233#file-0-optimize-import-patch >> > This patch moves a few imports inside functions. I wonder whether that kind > of change actually helps with real applications—doesn't any real application > end up importing the socket module anyway at some point? I don't know if this particular change is worthwhile, but one place where startup slowness is particularly noticed is with commands like 'foo.py --help' or 'foo.py --generate-completions' (the latter called implicitly by hitting in some shell), which typically do lots of imports that end up not being used. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
On 9/5/2017 2:20 PM, INADA Naoki wrote: On Wed, Sep 6, 2017 at 2:13 AM, Eric Snow wrote: Like Antoine, I consider the pure Python implementation to be valuable. Furthermore, the pure Python implementation is the reference, so its behavior is idiomatic. To this native English speaker, 'behavior is idiomatic' is not idiomatic ;-). The word 'idiomatic' refers to mode of expression, not mechanical behavior. Perhaps what Eric is saying is that the behavior of the reference implementation is by definition correct. Yet we have learned over the years to be more careful about what cpython behaviors constitute 'python' and which are incidental implementations. 'Improving the test suite' sometimes mean segregating the two. What is *idiomatic*? Expression that seems natural to a native speaker of a language, dialect, or subculture. The question you address is which behaviors of a Python OrderedDict are part of the definition and which are implementation-specific, and whether the tests need improvement. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Workflow change reminder: The Blurb Has Landed
Yesterday I "blurbified" the 2.7, 3.6, and master branches in CPython. It's finally official: all* branches have now been "blurbified". This means that Misc/NEWS is no longer present in any of CPython's active branches. Instead, Misc/NEWS has been broken up into a zillion little files in the new Misc/NEWS.d directory tree. (Misc/NEWS can be reconstituted consistently from this directory tree.) This banishes forever the annoying problem of Misc/NEWS merge conflicts when you attempt to merge a pull request, when release managers create a release, etc. We announced "blurb" a month or two back, and you should already be creating your news entries in Misc/NEWS.d. But just in case, here's a quick refresher. The core-workflow team has created a tool called "blurb" that makes it easy to create a news entry for a commit. It's easy to install and use. You can install it with: python3 -m pip install blurb It requires Python 3.5 or newer. (Sorry, it can't get backported to 3.4 without fixing a bug in "re"... and 3.4 is closed for bugfixes.) To create a news entry, just run blurb from anywhere inside your CPython repo. On POSIX systems just make sure "blurb" is on your path, then simply run "blurb". On Windows the recommended method is "python3 -m blurb". When you run blurb, it'll guide you through creating a news entry. It'll pop open an editor window on a template. Just modify the template as needed, write your news entry at the bottom, and save and exit. blurb will write out your news entry as a uniquely-named file in Misc/NEWS.d--and even stage it for you in "git"! If for some reason you want to create news entries by hand, or if you just want more information, please see the Dev Guide: https://devguide.python.org/committing/#what-s-new-and-news-entries and the documentation for blurb on PyPI: https://pypi.org/project/blurb/ The migration to Misc/NEWS.d and blurb has a few other implications: 1. Existing pending PRs or patches with Misc/NEWS entries will need to be updated to generate the entry with blurb. (Again, there really shouldn't /be/ any by this point.) 2. If you want to read a branch's NEWS entries in a convenient format, simply use "blurb merge" to produce a temporary NEWS file. (But don't check it in!) See the blurb documentation. 3. As part of the release process, release managers now use blurb to merge all the individual NEWS.d/next files into a single NEWS.d file for that version. They'll also build a traditional monolithic Misc/NEWS included in the release's source tarball (but this won't get checked in). 4. If you're building 3.x docs from a cpython git repo, you'll now need to have blurb installed. You can use the "make venv" recipe in Doc/Makefile to create a python3 venv with blurb and sphinx. Happy blurbing! /arry * All except 3.3. Ned Deily has taken over as release manager of 3.3, and he thinks we shouldn't bother with blurbifying that branch--it'll reach end-of-life in mere weeks, and we're only going to make one more release of it, and it gets very little work done these days. p.s. Thanks to Ned Deily who originally wrote this email, which I hacked up and sent. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 549: Instance Properties (aka: module properties)
I've written a PEP proposing a language change: https://www.python.org/dev/peps/pep-0549/ The TL;DR summary: add support for property objects to modules. I've already posted a prototype. How's that sound? //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)
On Tue, Sep 5, 2017 at 3:03 PM, Larry Hastings wrote: > > I've written a PEP proposing a language change: > > https://www.python.org/dev/peps/pep-0549/ > > The TL;DR summary: add support for property objects to modules. I've > already posted a prototype. Interesting idea! It's definitely less arcane than the __class__ assignment support that was added in 3.5. I guess the question is whether to add another language feature here, or to provide better documentation/helpers for the existing feature. If anyone's curious what the __class__ trick looks like in practice, here's some simple deprecation machinery: https://github.com/njsmith/trio/blob/ee8d909e34a2b28d55b5c6137707e8861eee3234/trio/_deprecate.py#L102-L138 And here's what it looks like in use: https://github.com/njsmith/trio/blob/ee8d909e34a2b28d55b5c6137707e8861eee3234/trio/__init__.py#L91-L115 Advantages of PEP 549: - easier to explain and use Advantages of the __class__ trick: - faster (no need to add an extra step to the normal attribute lookup fast path); only those who need the feature pay for it - exposes the full power of Python's class model. Notice that the above code overrides __getattr__ but not __dir__, so the attributes are accessible via direct lookup but not listed in dir(mod). This is on purpose, for two reasons: (a) tab completion shouldn't be suggesting deprecated attributes, (b) when I did expose them in __dir__, I had trouble with test runners that iterated through dir(mod) looking for tests, and ended up spewing tons of spurious deprecation warnings. (This is especially bad when you have a policy of running your tests with DeprecationWarnings converted to errors.) I don't think there's any way to do this with PEP 549. - already supported in CPython 3.5+ and PyPy3, and with a bit of care can be faked on all older CPython releases (including, crucially, CPython 2). PEP 549 OTOH AFAICT will only be usable for packages that have 3.7 as their minimum supported version. I don't imagine that I would actually use PEP 549 any time in the foreseeable future, due to the inability to override __dir__ and the minimum version requirement. If you only need to support CPython 3.5+ and PyPy3 5.9+, then you can effectively get PEP 549's functionality at the cost of 3 lines of code and a block indent: import sys, types class _MyModuleType(types.ModuleType): @property def ... @property def ... sys.modules[__name__].__class__ = _MyModuleType It's definitely true though that they're not the most obvious lines of code :-) -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v4
Yury Selivanov wrote: Question: how to write a context manager with contextvar.new? var = new_context_var() class CM: def __enter__(self): var.new(42) with CM(): print(var.get() or 'None') My understanding that the above code will print "None", because "var.new()" makes 42 visible only to callees of __enter__. If you tie the introduction of a new scope for context vars to generators, as PEP 550 currently does, then this isn't a problem. But I'm trying to avoid doing that. The basic issue is that, ever since yield-from, "generator" and "task" are not synonymous. When you use a generator to implement an iterator, you probably want it to behave as a distinct task with its own local context. But a generator used with yield-from isn't a task of its own, it's just part of another task, and there is nothing built into Python that lets you tell the difference automatically. So I'm now thinking that the introduction of a new local context should also be explicit. Suppose we have these primitives: push_local_context() pop_local_context() Now introducing a temporary decimal context looks like: push_local_context() decimal.localcontextvar.new(decimal.getcontext().copy()) decimal.localcontextvar.prec = 5 do_some_calculations() pop_local_context() Since calls (either normal or generator) no longer automatically result in a new local context, we can easily factor this out into a context manager: class LocalDecimalContext(): def __enter__(self): push_local_context() ctx = decimal.getcontext().copy() decimal.localcontextvar.new(ctx) return ctx def __exit__(self): pop_local_context() Usage: with LocalDecimalContext() as ctx: ctx.prec = 5 do_some_calculations() -- Greg But if I use "set()" in "CM.__enter__", presumably, it will traverse the stack of LCs to the very bottom and set "var=42" in in it. Right? If so, how can fix the example in PEP 550 Rationale: https://www.python.org/dev/peps/pep-0550/#rationale where we zip() the "fractions()" generator? With current PEP 550 semantics that's trivial: https://www.python.org/dev/peps/pep-0550/#generators Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 548: More Flexible Loop Control
I've written a PEP proposing a small enhancement to the Python loop control statements. Short version: here's what feels to me like a Pythonic way to spell "repeat until": while: break if The PEP goes into some detail on why this feels like a readability improvement in the more general case, with examples taken from the standard library: https://www.python.org/dev/peps/pep-0548/ Unlike Larry, I don't have a prototype, and in fact if this idea meets with approval I'll be looking for a volunteer to do the actual implementation. --David PS: this came to me in a dream on Sunday night, and the more I explored the idea the better I liked it. I have no idea what I was dreaming about that resulted in this being the thing left in my mind when I woke up :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v4
On Tue, Sep 5, 2017 at 4:59 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> >> Question: how to write a context manager with contextvar.new? >> >> var = new_context_var() >> >>class CM: >> >> def __enter__(self): >> var.new(42) >> >>with CM(): >> print(var.get() or 'None') >> >> My understanding that the above code will print "None", because >> "var.new()" makes 42 visible only to callees of __enter__. > > > If you tie the introduction of a new scope for context vars > to generators, as PEP 550 currently does, then this isn't a > problem. > > But I'm trying to avoid doing that. The basic issue is that, > ever since yield-from, "generator" and "task" are not > synonymous. > > When you use a generator to implement an iterator, you > probably want it to behave as a distinct task with its > own local context. But a generator used with yield-from > isn't a task of its own, it's just part of another task, > and there is nothing built into Python that lets you > tell the difference automatically. Greg, have you seen this new section: https://www.python.org/dev/peps/pep-0550/#should-yield-from-leak-context-changes ? It has a couple of examples that illustrate some issues with the "But a generator used with yield-from isn't a task of its own, it's just part of another task," reasoning. In principle, we can modify PEP 550 to make 'yield from' transparent to context changes. The interpreter can just reset g.__logical_context__ to None whenever 'g' is being 'yield-frommed'. The key issue is that there are a couple of edge-cases when having this semantics is problematic. The bottomline is that it's easier to reason about context when it's guaranteed that context changes are always isolated in generators no matter what. I think this semantics actually makes the refactoring easier. Please take a look at the linked section. > > So I'm now thinking that the introduction of a new local > context should also be explicit. > > Suppose we have these primitives: > >push_local_context() >pop_local_context() > > Now introducing a temporary decimal context looks like: > >push_local_context() >decimal.localcontextvar.new(decimal.getcontext().copy()) >decimal.localcontextvar.prec = 5 >do_some_calculations() >pop_local_context() > > Since calls (either normal or generator) no longer automatically > result in a new local context, we can easily factor this out into > a context manager: > >class LocalDecimalContext(): > > def __enter__(self): > push_local_context() > ctx = decimal.getcontext().copy() > decimal.localcontextvar.new(ctx) > return ctx > > def __exit__(self): > pop_local_context() > > Usage: > >with LocalDecimalContext() as ctx: > ctx.prec = 5 > do_some_calculations() This will have some performance implications and make the API way more complex. But I'm not convinced yet that real-life code needs the semantics you want. This will work with the current PEP 550 design: def g(): with DecimalContext() as ctx: ctx.prec = 5 yield from do_some_calculations() # will run with the correct ctx the only thing that won't work is this: def do_some_calculations(): ctx = DecimalContext() ctx.prec = 10 decimal.setcontext(ctx) yield def g(): yield from do_some_calculations() # Context changes in do_some_calculations() will not leak to g() In the above example, do_some_calculations() deliberately tries to leak context changes (by not using a contextmanager). And I consider it a feature that PEP 550 does not allow generators to leak state. If you write code that uses 'with' statements consistently, you will never even know that context changes are isolated in generators. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cherry picker bot deployed in CPython repo
Hi, The cherry picker bot has just been deployed to CPython repo, codenamed miss-islington. miss-islington made the very first backport PR for CPython and became a first time GitHub contributor: https://github.com/python/cpython/pull/3369 GitHub repo: https://github.com/python/miss-islington What is this? == As part of our workflow, quite often changes made on the master branch need to be backported to the earlier versions. (for example: from master to 3.6 and 2.7) Previously the backport has to be done manually by either a core developer or the original PR author. With the bot, the backport PR is created automatically after the PR has been merged. A core developer will need to review the backport PR. The issue was tracked in https://github.com/python/core-workflow/issues/8 How it works == 1. If a PR needs to be backported to one of the maintenance branches, a core developer should apply the "needs backport to X.Y" label. Do this **before** you merge the PR. 2. Merge the PR 3. miss-islington will leave a comment on the PR, saying it is working on backporting the PR. 4. If there's no merge conflict, the PR should be created momentarily. 5. Review the backport PR created by miss-islington and merge it when you're ready. Merge Conflicts / Problems? == In case of merge conflicts, or if a backport PR was not created within 2 minutes, it likely failed and you should do the backport manually. Manual backport can be done using cherry_picker: https://pypi.org/project/cherry-picker/ Older merged PRs not yet backported? == At the moment, those need to be backported manually. Don't want PR to be backported by a bot? My recommendation is to apply the "needs backport to X.Y" **after** the PR has been merged. The label is still useful to remind ourselves that this PR still needs backporting. Who is Miss Islington? = I found out from Wikipedia that Miss Islington is the name of the witch in Monty Python and The Holy Grail. miss-islington has not signed the CLA! = A core dev can ignore the warning and merge the PR anyway. Thanks! Mariatta Wijaya ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 553: Built-in debug()
I’ve written a PEP proposing the addition of a new built-in function called debug(). Adding this to your code would invoke a debugger through the hook function sys.debughook(). Like the existing sys.displayhook() and sys.excepthook(), you can change sys.debughook() to point to the debugger of your choice. By default it invokes pdb.set_trace(). With this PEP instead of: foo() import pdb; pdb.set_trace() bar() you can write: foo() debug() bar() and you would drop into the debugger after foo() but before bar(). More rationale and details are provided in the PEP: https://www.python.org/dev/peps/pep-0553/ Unlike David, but like Larry, I have a prototype implementation: https://github.com/python/cpython/pull/3355 Cheers, -Barry P.S. This came to me in a nightmare on Sunday night, and the more I explored the idea the more it frightened me. I know exactly what I was dreaming about and the only way to make it all go away was to write this thing up. signature.asc Description: Message signed with OpenPGP ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Cherry picker bot deployed in CPython repo
This is a great UX win for our development process. Thanks for making this happen! Alex On Tue, Sep 5, 2017 at 9:10 PM, Mariatta Wijaya wrote: > Hi, > > The cherry picker bot has just been deployed to CPython repo, codenamed > miss-islington. > > miss-islington made the very first backport PR for CPython and became a > first time GitHub contributor: https://github.com/python/cpython/pull/3369 > > > GitHub repo: https://github.com/python/miss-islington > > What is this? > == > > As part of our workflow, quite often changes made on the master branch > need to be backported to the earlier versions. (for example: from master to > 3.6 and 2.7) > > Previously the backport has to be done manually by either a core developer > or the original PR author. > > With the bot, the backport PR is created automatically after the PR has > been merged. A core developer will need to review the backport PR. > > The issue was tracked in https://github.com/python/core-workflow/issues/8 > > How it works > == > > 1. If a PR needs to be backported to one of the maintenance branches, a > core developer should apply the "needs backport to X.Y" label. Do this > **before** you merge the PR. > > 2. Merge the PR > > 3. miss-islington will leave a comment on the PR, saying it is working on > backporting the PR. > > 4. If there's no merge conflict, the PR should be created momentarily. > > 5. Review the backport PR created by miss-islington and merge it when > you're ready. > > Merge Conflicts / Problems? > == > > In case of merge conflicts, or if a backport PR was not created within 2 > minutes, it likely failed and you should do the backport manually. > > Manual backport can be done using cherry_picker: https://pypi. > org/project/cherry-picker/ > > Older merged PRs not yet backported? > == > > At the moment, those need to be backported manually. > > Don't want PR to be backported by a bot? > > > My recommendation is to apply the "needs backport to X.Y" **after** the PR > has been merged. The label is still useful to remind ourselves that this PR > still needs backporting. > > Who is Miss Islington? > = > > I found out from Wikipedia that Miss Islington is the name of the witch in > Monty Python and The Holy Grail. > > miss-islington has not signed the CLA! > = > > A core dev can ignore the warning and merge the PR anyway. > > Thanks! > > > Mariatta Wijaya > > ___ > python-committers mailing list > python-committ...@python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] Cherry picker bot deployed in CPython repo
Congrats! INADA Naoki On Wed, Sep 6, 2017 at 10:10 AM, Mariatta Wijaya wrote: > Hi, > > The cherry picker bot has just been deployed to CPython repo, codenamed > miss-islington. > > miss-islington made the very first backport PR for CPython and became a > first time GitHub contributor: https://github.com/python/cpython/pull/3369 > > GitHub repo: https://github.com/python/miss-islington > > What is this? > == > > As part of our workflow, quite often changes made on the master branch need > to be backported to the earlier versions. (for example: from master to 3.6 > and 2.7) > > Previously the backport has to be done manually by either a core developer > or the original PR author. > > With the bot, the backport PR is created automatically after the PR has been > merged. A core developer will need to review the backport PR. > > The issue was tracked in https://github.com/python/core-workflow/issues/8 > > How it works > == > > 1. If a PR needs to be backported to one of the maintenance branches, a core > developer should apply the "needs backport to X.Y" label. Do this **before** > you merge the PR. > > 2. Merge the PR > > 3. miss-islington will leave a comment on the PR, saying it is working on > backporting the PR. > > 4. If there's no merge conflict, the PR should be created momentarily. > > 5. Review the backport PR created by miss-islington and merge it when you're > ready. > > Merge Conflicts / Problems? > == > > In case of merge conflicts, or if a backport PR was not created within 2 > minutes, it likely failed and you should do the backport manually. > > Manual backport can be done using cherry_picker: > https://pypi.org/project/cherry-picker/ > > Older merged PRs not yet backported? > == > > At the moment, those need to be backported manually. > > Don't want PR to be backported by a bot? > > > My recommendation is to apply the "needs backport to X.Y" **after** the PR > has been merged. The label is still useful to remind ourselves that this PR > still needs backporting. > > Who is Miss Islington? > = > > I found out from Wikipedia that Miss Islington is the name of the witch in > Monty Python and The Holy Grail. > > miss-islington has not signed the CLA! > = > > A core dev can ignore the warning and merge the PR anyway. > > Thanks! > > > Mariatta Wijaya > > ___ > python-committers mailing list > python-committ...@python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RFC] Removing pure Python implementation of OrderedDict
> >> For example, when one thread do `od[k1] = v1` and another thread do >> `od[k2] = v2`, >> result should be equivalent to one of `od[k1] = v1; od[k2] = v2;` or >> `od[k2] = v2; od[k1] = v1`. And internal data structure should be >> consistent. > > I agree the pure Python OrderedDict is not thread-safe. But who said > it should? No one. I meant just let's say "it should" from Python 3.7. > And, actually, are you sure the C implementation is? GIL switches can > happen at many points. A simple Py_DECREF, or perhaps even a tuple > allocation can trigger a GC run that may release the GIL at some point > in a finalizer function. > I know such difficulity already. I thought simple (when key is str or int) __setitem__ doesn't break internal state of current C implementation of OrderedDict. And I'll carefully review my code too. >> Sometime, writing reentrant and threadsafe container in C is easier than >> in Pure Python. > > The C and Python versions needn't make exactly the same guarantees. > Yes, of course. But what we can recommend for library developers (including stdlib)? Currently, dict ordering is implementation detail of CPython and PyPy. We don't recommend to rely on the behavior. Like that, should we say "atomic & threadsafe __setitem__ for simple key is implementation detail of CPython and PyPy. We recommend using mutex when using OrderedDict from multiple thread."? That was my point about thread safety. Regards, INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
> It's a lot to type (27 characters). True. Personally I have a shortcut in my IDE (Sublime) so I when I type "pdb" -> TAB it auto completes it. > Python linters (e.g. flake8 [1]) complain about this line because it contains two statements. Breaking the idiom up into two lines further complicates the use of the debugger, I see this more as an advantage. After all a pdb line is "temporary" and you never want to commit it. When you do it is by accident so you want it to be more noticeable. I can configure my IDE to use flake8 and highlight the pdb line for me so that I can immediately see it because it's not PEP-8 compliant. I can have a GIT commit hook which checks there's no "pdb.set_trace()" in the files I'm committing: https://github.com/giampaolo/psutil/blob/master/.git-pre-commit#L93-L96 Somehow I think debug() would make this a bit harder as it's more likely a "debug()" line will pass unnoticed. For this reason I would give a -1 to this proposal. > It ties debugging directly to the choice of pdb. There might be other debugging options, say if you're using an IDE or some other development environment. Personally I would find it helpful if there was a hook to choose the default debugger to use on "pdb.set_trace()" via .pdbrc or PYTHONDEBUGGER environment variable or something. I tried (unsuccessfully) to run ipdb on "pdb.set_trace()", I gave up and ended up emulating auto completion and commands history with this: https://github.com/giampaolo/sysconf/blob/master/home/.pdbrc.py On Wed, Sep 6, 2017 at 9:14 AM, Barry Warsaw wrote: > I’ve written a PEP proposing the addition of a new built-in function > called debug(). Adding this to your code would invoke a debugger through > the hook function sys.debughook(). > > Like the existing sys.displayhook() and sys.excepthook(), you can change > sys.debughook() to point to the debugger of your choice. By default it > invokes pdb.set_trace(). > > With this PEP instead of: > > foo() > import pdb; pdb.set_trace() > bar() > > you can write: > > foo() > debug() > bar() > > and you would drop into the debugger after foo() but before bar(). More > rationale and details are provided in the PEP: > > https://www.python.org/dev/peps/pep-0553/ > > Unlike David, but like Larry, I have a prototype implementation: > > https://github.com/python/cpython/pull/3355 > > Cheers, > -Barry > > P.S. This came to me in a nightmare on Sunday night, and the more I > explored the idea the more it frightened me. I know exactly what I was > dreaming about and the only way to make it all go away was to write this > thing up. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/g. > rodola%40gmail.com > > -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
On Tue, Sep 5, 2017 at 6:14 PM, Barry Warsaw wrote: > I’ve written a PEP proposing the addition of a new built-in function called > debug(). Adding this to your code would invoke a debugger through the hook > function sys.debughook(). The 'import pdb; pdb.set_trace()' dance is *extremely* obscure, so replacing it with something more friendly seems like a great idea. Maybe breakpoint() would be a better description of what set_trace() actually does? This would also avoid confusion with IPython's very useful debug magic: https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-debug and which might also be worth stealing for the builtin REPL. (Personally I use it way more often than set_trace().) Example: In [1]: def f(): ...: x = 1 ...: raise RuntimeError ...: In [2]: f() --- RuntimeError Traceback (most recent call last) in () > 1 f() in f() 1 def f(): 2 x = 1 > 3 raise RuntimeError RuntimeError: In [3]: debug > (3)f() 1 def f(): 2 x = 1 > 3 raise RuntimeError ipdb> p x 1 -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
On Tue, Sep 05, 2017 at 06:14:12PM -0700, Barry Warsaw wrote: > I’ve written a PEP proposing the addition of a new built-in function > called debug(). Adding this to your code would invoke a debugger > through the hook function sys.debughook(). [...] > P.S. This came to me in a nightmare on Sunday night, and the more I > explored the idea the more it frightened me. I know exactly what I > was dreaming about and the only way to make it all go away was to > write this thing up. Sorry, are we to interpret this as you asking that the PEP be rejected? I can't tell whether you are being poetic and actually think the PEP is a good idea, or whether you have written it to have it rejected and prevent anyone else ever implementing this? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
On Tue, Sep 5, 2017, at 19:57, Steven D'Aprano wrote: > Sorry, are we to interpret this as you asking that the PEP be rejected? > I can't tell whether you are being poetic and actually think the PEP is > a good idea, or whether you have written it to have it rejected and > prevent anyone else ever implementing this? It is a playful reference to the postscript on the earlier https://mail.python.org/pipermail/python-dev/2017-September/149174.html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
Yeah, I like the idea, but I don't like the debug() name -- IIRC there's a helper named debug() in some codebase I know of that prints its arguments under certain circumstances. On Tue, Sep 5, 2017 at 7:58 PM, Nathaniel Smith wrote: > On Tue, Sep 5, 2017 at 6:14 PM, Barry Warsaw wrote: > > I’ve written a PEP proposing the addition of a new built-in function > called debug(). Adding this to your code would invoke a debugger through > the hook function sys.debughook(). > > The 'import pdb; pdb.set_trace()' dance is *extremely* obscure, so > replacing it with something more friendly seems like a great idea. > > Maybe breakpoint() would be a better description of what set_trace() > actually does? This would also avoid confusion with IPython's very > useful debug magic: > https://ipython.readthedocs.io/en/stable/interactive/ > magics.html#magic-debug > and which might also be worth stealing for the builtin REPL. > (Personally I use it way more often than set_trace().) > > Example: > > In [1]: def f(): >...: x = 1 >...: raise RuntimeError >...: > > In [2]: f() > > --- > RuntimeError Traceback (most recent call last) > in () > > 1 f() > > in f() > 1 def f(): > 2 x = 1 > > 3 raise RuntimeError > > RuntimeError: > > In [3]: debug > > (3)f() > 1 def f(): > 2 x = 1 > > 3 raise RuntimeError > > ipdb> p x > 1 > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Looking for review on small argparse patch
As argparse seems to not have a current maintainer, I'm hoping a volunteer could review this small patch to the argparse module. I posted the PR nearly a month ago and have not received any feedback yet: https://github.com/python/cpython/pull/3027 bpo link: https://bugs.python.org/issue26510 Thanks in advance, Anthony ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
>> >> I haven't created pull request yet. >> (Can I create without issue, as trivial patch?) > > > Trivial, no-issue PRs are meant for things like typo fixes that need no > discussion or record. > > Moving imports in violation of the PEP 8 rule, "Imports are always put at > the top of the file, just after any module comments and docstrings, and > before module globals and constants", is not trivial. Doing so voluntarily > for speed, as opposed to doing so necessarily to avoid circular import > errors, is controversial. > > -- > Terry Jan Reedy > Make sense. I'll create issues for each module if it seems really worth enough. Thanks, ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
>> With this profile, I tried optimize `python -c 'import asyncio'`, logging >> and http.client. >> >> >> https://gist.github.com/methane/1ab97181e74a33592314c7619bf34233#file-0-optimize-import-patch >> > This patch moves a few imports inside functions. I wonder whether that kind > of change actually helps with real applications—doesn't any real application > end up importing the socket module anyway at some point? > Ah, I'm sorry. It doesn't importing asyncio, logging and http.client faster. I saw pkg_resources. While it's not stdlib, it is imported very often. And it uses email.parser, but doesn't require socket or random. Since socket module creates some enums, removing it reduces few milliseconds. Regards, ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
>> This patch moves a few imports inside functions. I wonder whether that kind >> of change actually helps with real applications—doesn't any real application >> end up importing the socket module anyway at some point? > > I don't know if this particular change is worthwhile, but one place > where startup slowness is particularly noticed is with commands like > 'foo.py --help' or 'foo.py --generate-completions' (the latter called > implicitly by hitting in some shell), which typically do lots of > imports that end up not being used. > Yes. And There are more worse scenario. 1. Jinja2 supports asyncio. So it imports asyncio. 2. asyncio imports concurrent.futures, for compatibility with Future class. 3. concurrent.futures package does `from concurrent.futures.process import ProcessPoolExecutor` 4. concurrent.futures.process package imports multiprocessing. So when I use Jinja2 but not asyncio or multiprocessing, I need to import large dependency tree. I want to make `import asyncio` dependency tree smaller. FYI, current version of Jinja2 has very large regex which took more than 100ms when import time. It is fixed in master branch. So if you try to see Jinja2, please use master branch. Regrads, ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To reduce Python "application" startup time
On Wed, Sep 6, 2017 at 2:30 PM, INADA Naoki wrote: >>> This patch moves a few imports inside functions. I wonder whether that kind >>> of change actually helps with real applications—doesn't any real application >>> end up importing the socket module anyway at some point? >> >> I don't know if this particular change is worthwhile, but one place >> where startup slowness is particularly noticed is with commands like >> 'foo.py --help' or 'foo.py --generate-completions' (the latter called >> implicitly by hitting in some shell), which typically do lots of >> imports that end up not being used. >> > > Yes. And There are more worse scenario. > > 1. Jinja2 supports asyncio. So it imports asyncio. > 2. asyncio imports concurrent.futures, for compatibility with Future class. > 3. concurrent.futures package does > `from concurrent.futures.process import ProcessPoolExecutor` > 4. concurrent.futures.process package imports multiprocessing. > > So when I use Jinja2 but not asyncio or multiprocessing, I need to import > large dependency tree. > I want to make `import asyncio` dependency tree smaller. > > FYI, current version of Jinja2 has very large regex which took more than 100ms > when import time. It is fixed in master branch. So if you try to see > Jinja2, please > use master branch. How significant is application startup time to something that uses Jinja2? Are there short-lived programs that use it? Python startup time matters enormously to command-line tools like Mercurial, but far less to something that's designed to start up and then keep running (eg a web app, which is where Jinja is most used). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 548: More Flexible Loop Control
On Wed, Sep 6, 2017 at 10:11 AM, R. David Murray wrote: > I've written a PEP proposing a small enhancement to the Python loop > control statements. Short version: here's what feels to me like a > Pythonic way to spell "repeat until": > > while: > > break if > > The PEP goes into some detail on why this feels like a readability > improvement in the more general case, with examples taken from > the standard library: > > https://www.python.org/dev/peps/pep-0548/ Is "break if" legal in loops that have their own conditions as well, or only in a bare "while:" loop? For instance, is this valid? while not found_the_thing_we_want: data = sock.read() break if not data process(data) Or this, which uses the condition purely as a descriptor: while "moar socket data": data = sock.read() break if not data process(data) Also - shouldn't this be being discussed first on python-ideas? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 548: More Flexible Loop Control
06.09.17 03:11, R. David Murray пише: I've written a PEP proposing a small enhancement to the Python loop control statements. Short version: here's what feels to me like a Pythonic way to spell "repeat until": while: break if The PEP goes into some detail on why this feels like a readability improvement in the more general case, with examples taken from the standard library: https://www.python.org/dev/peps/pep-0548/ Unlike Larry, I don't have a prototype, and in fact if this idea meets with approval I'll be looking for a volunteer to do the actual implementation. --David PS: this came to me in a dream on Sunday night, and the more I explored the idea the better I liked it. I have no idea what I was dreaming about that resulted in this being the thing left in my mind when I woke up :) This looks rather like Perl way than Python way. "There should be one-- and preferably only one --obvious way to do it." This proposing saves just a one line of the code. But it makes "break" and "continue" statement less visually distinguishable as it is seen in your example from uuid.py. If allow "break if" and "continue if", why not allow "return if"? Or arbitrary statement before "if"? This adds PHP-like inconsistency in the language. Current idiom is easier for modification. If you add the second condition, it may be that you need to execute different code before "break". while True: if not : break if not : break It is easy to modify the code with the current syntax, but the code with the proposed syntax should be totally rewritten. Your example from sre_parse.py demonstrates this. Please note that pre-exit code is slightly different. In the first case self.error() is called with one argument, and in the second case it is called with two arguments. Your rewritten code is not equivalent to the existing one. Other concern is that the current code is highly optimized for common cases. Your rewritten code checks the condition "c is None" two times in common case. I'm -1 for this proposition. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
