Re: [Python-Dev] Language Summit notes
On Thu, 17 Apr 2014 01:23:13 -0400, Terry Reedy tjre...@udel.edu wrote: On 4/16/2014 6:26 PM, Antoine Pitrou wrote: AP exams are starting to allow Python, but it's 10% of the AP CS exams. AP? (I thought that was me, but it sounds unlikely :-)) AP = Advanced Placement. US and Canadian high school students who have taken advanced (AP) courses equivalent to American college freshman courses can take AP exams to demonstrate that they learned and retained enough that they should get college credit for the course. https://en.wikipedia.org/wiki/Advanced_Placement I believe there is a committee for each subject that sets out a syllabus describing the subject that may be tested. The CS exam originally tested knowledge of Pascal (1984-1999) and switched to C++ (1999-2003) and then Java (2004-date). https://en.wikipedia.org/wiki/Advanced_Placement_Computer_Science#AP_Computer_Science_A The report is that Python is creeping in, though I an not sure exactly what the report above means. Python replacing Java as the AP CS language would be a fairly big deal here. As I understand it, there is a *new* (pilot?) comp-sci related AP test that is using Python. Jessica expects the existing compsci AP tests may move to Python at some point, but her timeframe was fairly long for that one. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] is the concept of 'reference ownership' no long applicable in Python 3.4?
Hi, I noticed the following changes in the C API manuals from 3.3.5 (and earlier versions) to 3.4. I don't know if these changes are deliberate and imply that we C extension developers no longer need to care about 'reference ownership' because of some improvements in 3.4. Could anyone clarify it? Thanks, Jianfeng --- 3.4 C API reference manual PyObjecthttps://docs.python.org/3/c-api/structures.html#c.PyObject* PySequence_GetItem(PyObjecthttps://docs.python.org/3/c-api/structures.html#c.PyObject *o, Py_ssize_t i)¶https://docs.python.org/3/c-api/sequence.html#c.PySequence_GetItem Return the ith element of o, or NULL on failure. This is the equivalent of the Python expression o[i]. PyObjecthttps://docs.python.org/3.4/c-api/structures.html#c.PyObject* PyList_GetItem(PyObjecthttps://docs.python.org/3.4/c-api/structures.html#c.PyObject *list, Py_ssize_t index) Return the object at position index in the list pointed to by list. The position must be positive, indexing from the end of the list is not supported. If index is out of bounds, return NULL and set an IndexErrorhttps://docs.python.org/3.4/library/exceptions.html#IndexError exception. --- 3.3.5 C API reference manual --- PyObjecthttps://docs.python.org/3.3/c-api/structures.html#PyObject* PySequence_GetItem(PyObjecthttps://docs.python.org/3.3/c-api/structures.html#PyObject *o, Py_ssize_t i)¶https://docs.python.org/3.3/c-api/sequence.html#PySequence_GetItem Return value: New reference. Return the ith element of o, or NULL on failure. This is the equivalent of the Python expression o[i]. PyObjecthttps://docs.python.org/3.3/c-api/structures.html#PyObject* PyList_GetItem(PyObjecthttps://docs.python.org/3.3/c-api/structures.html#PyObject *list, Py_ssize_t index) Return value: Borrowed reference. Return the object at position index in the list pointed to by list. The position must be positive, indexing from the end of the list is not supported. If index is out of bounds, return NULL and set an IndexErrorhttps://docs.python.org/3.3/library/exceptions.html#IndexError exception. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Wed Apr 16 2014 at 4:53:25 PM, Terry Reedy tjre...@udel.edu wrote:
On Wednesday, April 16, 2014 2:57:35 PM, Terry Reedy tjre...@udel.edu
mailto:tjre...@udel.edu wrote:
PS. In the user process sys.modules, there are numerous null
entries like these:
sys.modules['idlelib.os']
sys.modules['idlelib.tokenize'__]
sys.modules['idlelib.io http://idlelib.io']
etcetera
On 4/16/2014 3:10 PM, Dr. Brett Cannon wrote:
Is this Python 2 or 3?
Py 2. I should have said so. The entries do not appear in py3.
In Python 2 it means an attempt to perform a relative import failed but
an absolute in succeeded, e.g. from idlelib you imported os, so import
tried idlelib.os and then os.
*I* have not done anything. For tokenize, for instance, the existing
code just does what I though were absolute imports, in 2 files.
import tokenize
That's not an absolute import if it's within a package and you didn't
declare `from __future__ import absolute_import`.
Perhaps the extra entries have something to do with the fact that these
startup imports are invisible to user code, just like those done by the
interpreter itself on startup. 2.7 uses spawnv (and 3.4 uses subprocces)
to run something like one of the following.
python -c __import__('idlelib.run').run.main(False)
python -c __import__('run').main(False)
Nope, it has to simply do with how Python 2 does implicit relative imports.
Add the __future__ statement and they will go away.
-Brett
run.py has several normal lines with
import stdlib module
from idlelib import idlelib module
and ditto for some of the imported idlelib modules.
You should definitely consider using a future import to guarantee
absolute imports.
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
brett%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Language Summit notes
On Wed, Apr 16, 2014 at 11:26 PM, Antoine Pitrou solip...@pitrou.netwrote: What does this mean exactly? Under OS X and Linux, Python is typically installed by default. Under OS X, at least, I think there are valid reasons to not want to use the system-supplied Python. On my up-to-date OS X 10.9.2 machine, I see Python 2.7.5, NumPy 1.6.2, Matplotlib 1.1.1 and Twisted 12.2.0. For at least Matplotlib and NumPy, those versions are pretty old (mid 2012), and I'd be wary of updating them on the *system* Python: I have no idea what I might or might not break. -- Mark ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc
On Wed, Apr 16, 2014 at 12:51 PM, Julian Taylor jtaylor.deb...@googlemail.com wrote: Hi, In NumPy what we want is the tracing, not the exchangeable allocators. I don't think it is a good idea for the core of a whole stack of C-extension based modules to replace the default allocator or allowing other modules to replace the allocator NumPy uses. I don't think modules are ever supposed to replace the underlying allocator itself -- and it'd be very difficult to do this safely, since by the time any modules are imported there are already active allocations floating around. I think the allocator replacement functionality is designed to be used by applications embedding Python, who can set it up a special allocator before the interpreter starts. I'm not sure what exactly why one would need to swap out malloc and friends for something else, so I can't really judge, but it does at least seem plausible that if someone is taking the trouble to swap out the allocator like this then numpy should respect that and use the new allocator. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc
On Wed, Apr 16, 2014 at 7:35 PM, Victor Stinner
victor.stin...@gmail.com wrote:
Hi,
2014-04-16 7:51 GMT-04:00 Julian Taylor jtaylor.deb...@googlemail.com:
In NumPy what we want is the tracing, not the exchangeable allocators.
Did you read the PEP 445? Using the new malloc API, in fact you can
have both: install new allocators and set up hooks on allocators.
http://legacy.python.org/dev/peps/pep-0445/
The context here is that there's been some followup discussion on the
numpy list about whether there are cases where we need even more
exotic memory allocators than calloc(), and what to do about it if so.
(Thread: http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069935.html
)
One case that has come up is when efficient use of SIMD instructions
requires better-than-default alignment (e.g. malloc() usually gives
something like 8 byte alignment, but if you're using an instruction
that operates on 32 bytes at once you might need your array to have 32
byte alignment). Most (all?) OSes provide an extended version of
malloc that allows one to request more alignment (posix_memalign on
POSIX, _aligned_malloc on windows), and C11 standardizes this as
aligned_alloc. An important feature of these functions is that they
allocate from the same heap that 'malloc' does, i.e., when done with
the aligned memory you call free() -- there's no such thing as
aligned_free(). This means that if your program uses these functions
then swapping out malloc/free without swapping out aligned_alloc will
produce undesireable results.
Numpy does not currently use aligned allocation, and it's not clear
how important it is -- on older x86 it matters, but not so much on
current CPUs, but when the next round of x86 SIMD instructions are
released next year it might matter again, and apparently on popular
IBM supercomputers it matters (but less on newer versions)[1,2], and
who knows what will happen with ARM. It's a bit of a mess. But if
we're messing about with APIs it seems worth thinking about.
[1] http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069965.html
[2] http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069967.html
A second possible use case is:
my_hugetlb_alloc(size)
p = mmap('hugepagefs', ..., MAP_HUGETLB);
PyMem_Register_Alloc(p, size, __func__, __line__);
return p
my_hugetlb_free(p);
PyMem_Register_Free(p, __func__, __line__);
munmap(p, ...);
This is exactly how tracemalloc works. The advantage of the PEP 445 is
that you have a null overhead when tracemalloc is disabled. There is
no need to check if a trace function is present or not.
I think the key thing about this example is that you would *never*
want to use MAP_HUGETLB as a generic replacement for malloc(). Huge
pages can have all kinds of weird quirky limitations, and are
certainly unsuited for small allocations. BUT they can provide huge
speed wins if used for certain specific allocations in certain
programs. (In case anyone needs a reminder what huge pages even are:
http://lwn.net/Articles/374424/)
If I wrote a Python library to make it easy to use huge pages with
numpy, then I might well want the allocations I was making to be
visible to tracemalloc, even though they would not be going through
malloc/free.
(For that matter -- should calls to os.mmap be calling some
tracemalloc hook in general? There are lots of cases where mmap is
really doing memory allocation -- it's very useful for shared memory
and stuff too.)
---
My current impression is something like:
- From the bug report discussion it sounds like calloc() is useful
even in core Python, so it makes sense to go ahead with that
regardless.
- Now that aligned_alloc has been standardized, it might make sense to
add it to the PyMemAllocator struct too.
- And it might also make sense to have an API by which a Python
library can say to tracemalloc: hey FYI I just allocated something
using my favorite weird exotic method, like in the huge pages
example. This is a fully generic mechanism, so it could act as a kind
of safety valve for future weirdnesses.
All numpy *needs* to support its current and immediately foreseeable
usage is calloc(). But I'm a bit nervous about getting trapped -- if
the PyMem_* machinery implements calloc(), and we switch to using it
and advertise tracemalloc support to our users, and then later it
turns out that we need aligned_alloc or similar, then we'll be stuck
unless and until we can get at least one of these other changes into
CPython upstream, and that will suck for all of us.
--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is the concept of 'reference ownership' no long applicable in Python 3.4?
On Thu, Apr 17, 2014 at 4:34 PM, Jianfeng Mao j...@rocketsoftware.comwrote: I noticed the following changes in the C API manuals from 3.3.5 (and earlier versions) to 3.4. I don’t know if these changes are deliberate and imply that we C extension developers no longer need to care about ‘reference ownership’ because of some improvements in 3.4. Could anyone clarify it? AFAIK there's been no deliberate change to the notion of reference ownership. Moreover, any such change would break existing C extensions, so it's highly unlikely that anything's changed here, behaviour-wise. This looks like a doc build issue: when I build the documentation locally for the default branch, I still see the expected Return value: New reference. lines. Maybe something went wrong with refcounts.dat or the Sphinx refcounting extension when building the 3.4 documentation? Larry: any ideas? Mark ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] is the concept of 'reference ownership' no long applicable in Python 3.4?
On Thu, Apr 17, 2014 at 5:33 PM, Mark Dickinson dicki...@gmail.com wrote: This looks like a doc build issue: when I build the documentation locally for the default branch, I still see the expected Return value: New reference. lines. Opened http://bugs.python.org/issue21286 for this issue. -- Mark ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 - default): Merge 3.4
On Thu, Apr 17, 2014, at 8:55, matthias.klose wrote: http://hg.python.org/cpython/rev/1d1aefd00f07 changeset: 90382:1d1aefd00f07 parent: 90380:517de1983677 parent: 90381:1a00e04a233d user:d...@ubuntu.com date:Thu Apr 17 17:55:03 2014 +0200 summary: Merge 3.4 files: Misc/NEWS | 6 ++ setup.py | 15 +-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -244,8 +244,14 @@ Build - + local - Issue #17861: Tools/scripts/generate_opcode_h.py automatically regenerates Include/opcode.h from Lib/opcode.py if the later gets any change. +=== +- Issue #15234: For BerkelyDB and Sqlite, only add the found library and + include directories if they aren't already being searched. This avoids + an explicit runtime library dependency. + other You might want to actually resolve this conflict. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Not considering the zipped stdlib technique mentioned in other posts, would it perhaps be better to support two different CPython builds: - one with all the needed stdlib parts frozen - to be used in production - one with only the minimal needed number of stdlib parts frozen - to have as much of the stdlib sources readily accessible to application developers as possible The installer could then perhaps install both executables, or the frozen stdlib parts could perhaps be built as a separate DLL to be loaded at runtime instead of its content being used from their Python sources. OK... just my 2 cents worth... :-) Best regards, Jurko Gospodnetić ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'd like to understand what other situations occur frequently enough to worry about it. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Freezing modules does not affect the ability to use gdb. And as long as you set the appropriate __file__ values then tracebacks will contain even the file line and location. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). 2014-04-17 14:09 GMT-04:00 Brett Cannon bcan...@gmail.com: On Thu Apr 17 2014 at 1:34:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 14.4.2014. 23:51, Brett Cannon wrote: Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Freezing modules does not affect the ability to use gdb. And as long as you set the appropriate __file__ values then tracebacks will contain even the file line and location. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/marky1991%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Freezing modules does not affect the ability to use gdb. And as long as you set the appropriate __file__ values then tracebacks will contain even the file line and location. I sympathize with Jurko's opinion. Being able to poke inside stdlib source files makes Python more approachable. I'm sure several of us got into Python that way. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
I still do that! On Thu, Apr 17, 2014 at 11:17 AM, Antoine Pitrou solip...@pitrou.netwrote: On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Freezing modules does not affect the ability to use gdb. And as long as you set the appropriate __file__ values then tracebacks will contain even the file line and location. I sympathize with Jurko's opinion. Being able to poke inside stdlib source files makes Python more approachable. I'm sure several of us got into Python that way. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Hi. On 17.4.2014. 20:15, Mark Young wrote: I think he meant modifying the source files themselves for debugging purposes (e.g. putting print statements in itertools.py). Exactly! :-) Best regards, Jurko Gospodnetić ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On 04/17/2014 10:33 AM, Jurko Gospodnetić wrote: In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. +1 -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] List vs Tuple / Homogeneous vs Heterogeneous / Mutable vs Immutable
Hello there! I've stumbled upon this discussion on python-dev about what the choice between using a list or a tuple is all about in 2003: 1. https://mail.python.org/pipermail/python-dev/2003-March/033962.html 2. https://mail.python.org/pipermail/python-dev/2003-March/034029.html There's a vague comment about it on python documentation but afaik there the discussion hasn't made into any PEPs. Is there an understanding about it? Cheers, Leandro ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] this is what happens if you freeze all the modules required for startup
Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Avg: 0.521073 - 2.694876: 5.17x slower Significant (t=-1129.54) Stddev: 0.00478 - 0.01274: 2.6681x larger ### startup_nosite ### Min: 0.304090 - 0.908059: 2.99x slower Avg: 0.312374 - 0.921807: 2.95x slower Significant (t=-797.79) Stddev: 0.00372 - 0.00667: 1.7956x larger -Brett On Mon Apr 14 2014 at 5:51:23 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -- default vs the freezing: ### normal_startup ### Min: 0.524812 - 0.473339: 1.11x faster Avg: 0.534403 - 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 - 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 - 0.291939: 1.05x faster Avg: 0.317667 - 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 - 0.00385: 1.4099x smaller - 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 - 0.465264: 1.27x slower Avg: 0.374404 - 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 - 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 - 0.290544: 1.77x slower Avg: 0.169833 - 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 - 0.00407: 1.9310x larger - As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 - 0.521758: 1.41x slower Avg: 0.376784 - 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 - 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 - 0.309090: 1.87x slower Avg: 0.171516 - 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 - 0.00399: 1.1948x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu, Apr 17, 2014 at 8:17 PM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 17 Apr 2014 18:09:22 + Brett Cannon bcan...@gmail.com wrote: I would really love to have better startup times in production, but I would also really hate to lose the ability to hack around in stdlib sources during development just to get better startup performance. In general, what I really like about using Python for software development is the ability to open any stdlib file and easily go poking around using stuff like 'import pdb;pdb.set_trace()' or simple print statements. Researching mysterious behaviour is generally much much MUCH! easier (read: takes less hours/days/weeks) if it ends up leading into a stdlib Python module than if it takes you down into the bowels of some C module (think zipimport.c *grin*). Not to mention the effect that being able to quickly resolve a mystery by hacking on some Python internals leaves you feeling very satisfied, while having to entrench yourself in those internals for a long time just to find out you've made something foolish on your end leaves you feeling exhausted at best. Freezing modules does not affect the ability to use gdb. And as long as you set the appropriate __file__ values then tracebacks will contain even the file line and location. I sympathize with Jurko's opinion. Being able to poke inside stdlib source files makes Python more approachable. I'm sure several of us got into Python that way. Regards Antoine. I also wouldn't want that to be the default but Martin also suggested a -Z cmdline option which sounds like an interesting idea to me. ...Or maybe simply use the existent -O option, which doesn't really optimize much AFAIK. -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] List vs Tuple / Homogeneous vs Heterogeneous / Mutable vs Immutable
On Thu Apr 17 2014 at 2:43:35 PM, Leandro Pereira de Lima e Silva leandro...@cpti.cetuc.puc-rio.br wrote: Hello there! I've stumbled upon this discussion on python-dev about what the choice between using a list or a tuple is all about in 2003: 1. https://mail.python.org/pipermail/python-dev/2003-March/033962.html 2. https://mail.python.org/pipermail/python-dev/2003-March/034029.html There's a vague comment about it on python documentation but afaik there the discussion hasn't made into any PEPs. Is there an understanding about it? Think of tuples like a struct in C, lists like an array. That's just out of Guido's head so I don't think we have ever bothered to write it down somewhere as an important distinction of the initial design that should be emphasized. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Hi. On 17.4.2014. 19:57, Guido van Rossum wrote: On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr mailto:jurko.gospodne...@pke.hr wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'd like to understand what other situations occur frequently enough to worry about it. The first one that pops to mind is scripting when automating different system administration tasks. When you automate something that ends up calling lots of different Python scripts - the startup times add up. Yes, I know you can update the system so that the scripts get called inside a single Python process, but that often requires major refactoring, e.g.: - you have to refactor those scripts to be importable while they were originally prepared to be used as 'stand-alone executables' - you either have to use Python as your external automation tool or you need to implement some sort of a Python based tool runner daemon process Another example is the speed at which some automated test suits run that need to call external Python scripts. Such suites often call thousands of such scripts so their startup times add up to such numbers that Python gets a bad rep. And shaving off unnecessarily wasted seconds or minutes in a test suite is always good, as it speeds up the whole develop/test cycle. :-) I've been in situations where I got a request to 'convert those Python scripts to batch files so they would run faster'. :-) And, while I really love Python as a development language, simple scripts implemented in it often do make the system feel kind of sluggish. :-( And with that in mind, the effect of systems becoming 'even more sluggish' when upgrading them to use the new 'Python 3' version, even if that slowdown is not all startup related, often comes as an additional slap in the face. :-( Best regards, Jurko Gospodnetić ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] this is what happens if you freeze all the modules required for startup
Actually ignore this data, I think I may have messed something up. I'll reply after I check something On Thu Apr 17 2014 at 2:47:52 PM, Brett Cannon bcan...@gmail.com wrote: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Avg: 0.521073 - 2.694876: 5.17x slower Significant (t=-1129.54) Stddev: 0.00478 - 0.01274: 2.6681x larger ### startup_nosite ### Min: 0.304090 - 0.908059: 2.99x slower Avg: 0.312374 - 0.921807: 2.95x slower Significant (t=-797.79) Stddev: 0.00372 - 0.00667: 1.7956x larger -Brett On Mon Apr 14 2014 at 5:51:23 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -- default vs the freezing: ### normal_startup ### Min: 0.524812 - 0.473339: 1.11x faster Avg: 0.534403 - 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 - 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 - 0.291939: 1.05x faster Avg: 0.317667 - 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 - 0.00385: 1.4099x smaller - 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 - 0.465264: 1.27x slower Avg: 0.374404 - 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 - 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 - 0.290544: 1.77x slower Avg: 0.169833 - 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 - 0.00407: 1.9310x larger - As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 - 0.521758: 1.41x slower Avg: 0.376784 - 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 - 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 - 0.309090: 1.87x slower Avg: 0.171516 - 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 - 0.00399: 1.1948x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu Apr 17 2014 at 3:10:46 PM, Brett Cannon bcan...@gmail.com wrote: Actually ignore this data, I think I may have messed something up. I'll reply after I check something Unfortunately my check says the data is accurate, so zip startup is really just slow. -Brett On Thu Apr 17 2014 at 2:47:52 PM, Brett Cannon bcan...@gmail.com wrote: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Avg: 0.521073 - 2.694876: 5.17x slower Significant (t=-1129.54) Stddev: 0.00478 - 0.01274: 2.6681x larger ### startup_nosite ### Min: 0.304090 - 0.908059: 2.99x slower Avg: 0.312374 - 0.921807: 2.95x slower Significant (t=-797.79) Stddev: 0.00372 - 0.00667: 1.7956x larger -Brett On Mon Apr 14 2014 at 5:51:23 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -- default vs the freezing: ### normal_startup ### Min: 0.524812 - 0.473339: 1.11x faster Avg: 0.534403 - 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 - 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 - 0.291939: 1.05x faster Avg: 0.317667 - 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 - 0.00385: 1.4099x smaller - 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 - 0.465264: 1.27x slower Avg: 0.374404 - 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 - 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 - 0.290544: 1.77x slower Avg: 0.169833 - 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 - 0.00407: 1.9310x larger - As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 - 0.521758: 1.41x slower Avg: 0.376784 - 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 - 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 - 0.309090: 1.87x slower Avg: 0.171516 - 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 - 0.00399: 1.1948x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? On Thu, Apr 17, 2014 at 11:47 AM, Brett Cannon bcan...@gmail.com wrote: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Avg: 0.521073 - 2.694876: 5.17x slower Significant (t=-1129.54) Stddev: 0.00478 - 0.01274: 2.6681x larger ### startup_nosite ### Min: 0.304090 - 0.908059: 2.99x slower Avg: 0.312374 - 0.921807: 2.95x slower Significant (t=-797.79) Stddev: 0.00372 - 0.00667: 1.7956x larger -Brett On Mon Apr 14 2014 at 5:51:23 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -- default vs the freezing: ### normal_startup ### Min: 0.524812 - 0.473339: 1.11x faster Avg: 0.534403 - 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 - 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 - 0.291939: 1.05x faster Avg: 0.317667 - 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 - 0.00385: 1.4099x smaller - 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 - 0.465264: 1.27x slower Avg: 0.374404 - 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 - 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 - 0.290544: 1.77x slower Avg: 0.169833 - 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 - 0.00407: 1.9310x larger - As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 - 0.521758: 1.41x slower Avg: 0.376784 - 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 - 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 - 0.309090: 1.87x slower Avg: 0.171516 - 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 - 0.00399: 1.1948x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] List vs Tuple / Homogeneous vs Heterogeneous / Mutable vs Immutable
It's definitely something that should be put in some documentation, probably at the point when people have learned enough to be designing their own programs where this issue comes up -- before they're wizards but well after they have learned the semantic differences between lists and tuples. On Thu, Apr 17, 2014 at 11:49 AM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 2:43:35 PM, Leandro Pereira de Lima e Silva leandro...@cpti.cetuc.puc-rio.br wrote: Hello there! I've stumbled upon this discussion on python-dev about what the choice between using a list or a tuple is all about in 2003: 1. https://mail.python.org/pipermail/python-dev/2003-March/033962.html 2. https://mail.python.org/pipermail/python-dev/2003-March/034029.html There's a vague comment about it on python documentation but afaik there the discussion hasn't made into any PEPs. Is there an understanding about it? Think of tuples like a struct in C, lists like an array. That's just out of Guido's head so I don't think we have ever bothered to write it down somewhere as an important distinction of the initial design that should be emphasized. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Apr 17, 2014, at 2:23 PM, Jurko Gospodnetić jurko.gospodne...@pke.hr wrote: Hi. On 17.4.2014. 19:57, Guido van Rossum wrote: On Thu, Apr 17, 2014 at 10:33 AM, Jurko Gospodnetić jurko.gospodne...@pke.hr mailto:jurko.gospodne...@pke.hr wrote: I would really love to have better startup times in production, What's your use case? I understand why startup time is important for Hg, but I'd like to understand what other situations occur frequently enough to worry about it. The first one that pops to mind is scripting when automating different system administration tasks. When you automate something that ends up calling lots of different Python scripts - the startup times add up. Yes, I know you can update the system so that the scripts get called inside a single Python process, but that often requires major refactoring, e.g.: - you have to refactor those scripts to be importable while they were originally prepared to be used as 'stand-alone executables' - you either have to use Python as your external automation tool or you need to implement some sort of a Python based tool runner daemon process Another example is the speed at which some automated test suits run that need to call external Python scripts. Such suites often call thousands of such scripts so their startup times add up to such numbers that Python gets a bad rep. And shaving off unnecessarily wasted seconds or minutes in a test suite is always good, as it speeds up the whole develop/test cycle. :-) pip invokes a ton of pythons in a subprocess in it’s test suite, and the “install from sdist” stuff tends to invoke 1-3 python’s per thing you install too. So any speed up there would make installing stuff faster. I've been in situations where I got a request to 'convert those Python scripts to batch files so they would run faster'. :-) And, while I really love Python as a development language, simple scripts implemented in it often do make the system feel kind of sluggish. :-( And with that in mind, the effect of systems becoming 'even more sluggish' when upgrading them to use the new 'Python 3' version, even if that slowdown is not all startup related, often comes as an additional slap in the face. :-( Best regards, Jurko Gospodnetić ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] List vs Tuple / Homogeneous vs Heterogeneous / Mutable vs Immutable
No, I don't think it belongs in the style guide. It is not about code formatting or naming, it is about data structure design and API design. On Thu, Apr 17, 2014 at 12:49 PM, Leandro Pereira de Lima e Silva leandro...@cpti.cetuc.puc-rio.br wrote: This looks like an issue to be addressed at PEP-8 since it looks like a styling issue. I haven't seen any other recommendations there on how to use a certain data structure, though. Cheers, Leandro Em 17/04/2014 16:24, Guido van Rossum gu...@python.org escreveu: It's definitely something that should be put in some documentation, probably at the point when people have learned enough to be designing their own programs where this issue comes up -- before they're wizards but well after they have learned the semantic differences between lists and tuples. On Thu, Apr 17, 2014 at 11:49 AM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 2:43:35 PM, Leandro Pereira de Lima e Silva leandro...@cpti.cetuc.puc-rio.br wrote: Hello there! I've stumbled upon this discussion on python-dev about what the choice between using a list or a tuple is all about in 2003: 1. https://mail.python.org/pipermail/python-dev/2003-March/033962.html 2. https://mail.python.org/pipermail/python-dev/2003-March/034029.html There's a vague comment about it on python documentation but afaik there the discussion hasn't made into any PEPs. Is there an understanding about it? Think of tuples like a struct in C, lists like an array. That's just out of Guido's head so I don't think we have ever bothered to write it down somewhere as an important distinction of the initial design that should be emphasized. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum gu...@python.org wrote: I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? Startup where the stdlib is entirely in a zip file is slower than the status quo of reading from files. IOW it looks like speeding up startup from an import perspective requires either freezing modules -- for about a 10% boost -- or some fundamental change in import that no one has thought of yet. -Brett On Thu, Apr 17, 2014 at 11:47 AM, Brett Cannon bcan...@gmail.com wrote: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Avg: 0.521073 - 2.694876: 5.17x slower Significant (t=-1129.54) Stddev: 0.00478 - 0.01274: 2.6681x larger ### startup_nosite ### Min: 0.304090 - 0.908059: 2.99x slower Avg: 0.312374 - 0.921807: 2.95x slower Significant (t=-797.79) Stddev: 0.00372 - 0.00667: 1.7956x larger -Brett On Mon Apr 14 2014 at 5:51:23 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -- default vs the freezing: ### normal_startup ### Min: 0.524812 - 0.473339: 1.11x faster Avg: 0.534403 - 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 - 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 - 0.291939: 1.05x faster Avg: 0.317667 - 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 - 0.00385: 1.4099x smaller - 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 - 0.465264: 1.27x slower Avg: 0.374404 - 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 - 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 - 0.290544: 1.77x slower Avg: 0.169833 - 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 - 0.00407: 1.9310x larger - As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 - 0.521758: 1.41x slower Avg: 0.376784 - 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 - 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 - 0.309090: 1.87x slower Avg: 0.171516 - 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 - 0.00399: 1.1948x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu, Apr 17, 2014 at 1:31 PM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 3:21:49 PM, Guido van Rossum gu...@python.org wrote: I'm sorry to keep asking dumb questions, but your description didn't job my understanding of what you are comparing here. What is slower than what? Startup where the stdlib is entirely in a zip file is slower than the status quo of reading from files. That deserves more research. I'm not sure I believe we understand exactly what goes on in each case -- perhaps our zip reading code isn't as efficient as it could be? It would also be interesting to compare different platforms. IOW it looks like speeding up startup from an import perspective requires either freezing modules -- for about a 10% boost -- or some fundamental change in import that no one has thought of yet. And it's probably premature. (Unless you already have a prototype and it shows a solid speedup.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] List vs Tuple / Homogeneous vs Heterogeneous / Mutable vs Immutable
This looks like an issue to be addressed at PEP-8 since it looks like a styling issue. I haven't seen any other recommendations there on how to use a certain data structure, though. Cheers, Leandro Em 17/04/2014 16:24, Guido van Rossum gu...@python.org escreveu: It's definitely something that should be put in some documentation, probably at the point when people have learned enough to be designing their own programs where this issue comes up -- before they're wizards but well after they have learned the semantic differences between lists and tuples. On Thu, Apr 17, 2014 at 11:49 AM, Brett Cannon bcan...@gmail.com wrote: On Thu Apr 17 2014 at 2:43:35 PM, Leandro Pereira de Lima e Silva leandro...@cpti.cetuc.puc-rio.br wrote: Hello there! I've stumbled upon this discussion on python-dev about what the choice between using a list or a tuple is all about in 2003: 1. https://mail.python.org/pipermail/python-dev/2003-March/033962.html 2. https://mail.python.org/pipermail/python-dev/2003-March/034029.html There's a vague comment about it on python documentation but afaik there the discussion hasn't made into any PEPs. Is there an understanding about it? Think of tuples like a struct in C, lists like an array. That's just out of Guido's head so I don't think we have ever bothered to write it down somewhere as an important distinction of the initial design that should be emphasized. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). ### normal_startup ### Min: 0.510211 - 2.667958: 5.23x slower Not sure how to interpret this: what is 5.23x slower than what? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). [I found the answer on what is being compared in replies] So how did you create the zip file? Any chance that you may have compressed the pyc files? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
On Thu Apr 17 2014 at 5:21:14 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). [I found the answer on what is being compared in replies] Yeah, I did it in under 5 minutes on a whim so I wasn't entirely thinking when I posted the numbers. So how did you create the zip file? zip ../python35.zip -r . Any chance that you may have compressed the pyc files? Yes. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/17/2014 06:06 PM, Brett Cannon wrote: On Thu Apr 17 2014 at 5:21:14 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 17.04.14 20:47, schrieb Brett Cannon: Because people keep bringing it up, below is the results of hacking up the interpreter to include a sys.path entry for ./python35.zip instead of hard-coding to /usr/lib/python35.zip and simply zipped up Lib/ recursively. TL;DR, zipimport performance no longer measures up (probably because of stat caching and such that importlib introduced). [I found the answer on what is being compared in replies] Yeah, I did it in under 5 minutes on a whim so I wasn't entirely thinking when I posted the numbers. So how did you create the zip file? zip ../python35.zip -r . Any chance that you may have compressed the pyc files? I think you want 'zip -0' to avoid the compression. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlNQUzMACgkQ+gerLs4ltQ53XACcCihQVdb9h4RSnOphhkzu8AjU JsAAoJXClEcf4/McqA610Lh5SDdeHdhW =6pNL -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
