Re: [Python-Dev] Zipping the standard library.
On Sat, Mar 10, 2012 at 5:49 PM, Thomas Wouters wrote: > (And, yes, I'm zipping up the stdlib for Python 2.7 at Google, to reduce > the impact on the aforementioned million of machines :) > You might want to consider instead backporting the importlib caching facility, since it provides some of the zipimport benefits for plain old, non-zipped modules. Actually, a caching-only import hook that operated that way wouldn't even need the whole of importlib, just a wrapper over the standard C import that skips the unnecessary filesystem accesses. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Zipping the standard library.
On Sat, 10 Mar 2012 14:49:24 -0800 Thomas Wouters wrote: > Also, depending on what else you > want to put in the zipfile, you may have to be aware of zipimports limited > implementation of zipfiles that involve various 32k-filecount and > 2Gb-filesize limits. (And in case you're wondering, yes, we are doing this > with Python 2.7 at Google to save space. And yes, hitting the 2Gb limit is > quite possible for us.) Have you investigated filesystems with built-in compression? :) > Also, at that point the question becomes if we need a > transparent interface for opening module sourcefiles or arbitrary files > living in packages, that could grab things out of zipfiles (like setuptools > has in... one of the modules) -- or other archives of course. You mean pkg_resources I suppose. The problem is this kind of API is a PITA to use compared to the simple and obvious open() Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Zipping the standard library.
Since Python 2.3 (with the introduction of the zipimport module) it's been sort-of possible to zip up the standard library. Modules/getpath.c:calculate_path even adds a specific location ($prefix/lib/python33.zip) to sys.path if it exists to facilitate that. Or you can include the zipfile alongside an application that embeds Python, or even embed the zipfile in the same application. Actually setting things up is not quite that simple, though, at least on non-Windows: you need to know what to include in the zipfile (only .py, .pyc and .pyo files, no .so files), what to leave in the old location (os.py, or at least *some file called os.py*, needs to stay in $prefix/lib/python3.3) and how to deal with tests and modules that don't like the stdlib living in zipfiles -- and there's more of those than I expected. Also, depending on what else you want to put in the zipfile, you may have to be aware of zipimports limited implementation of zipfiles that involve various 32k-filecount and 2Gb-filesize limits. (And in case you're wondering, yes, we are doing this with Python 2.7 at Google to save space. And yes, hitting the 2Gb limit is quite possible for us.) So with importlib going in, should we do something with zipimport as well? Its deficiencies can easily be fixed by reimpementing it in Python instead -- the zipfile module has long since fixed the same 32k/2Gb issues (reading signed instead of unsigned numbers) and actually supports zip64 extensions (to break the 64k-filecount and 4Gb-filesize limits in "normal" zipfiles.) Actually supporting zipping the stdlib then becomes a bit harder: the importlib bootstrapping would need to include the zipfile module. If we do that, it would be nice to actually support zipping the stdlib in the Python build: making a build target that actually does that, and runs the tests with it. However, this requires modification of a whole bunch of tests, for example ones that assume that stdlib modules (and the tests themselves!) have actual files you can open() as their __file__ attribute, and we'd need to run the testsuite with the stdlib as a zip to prevent new ones from sneaking in. Also, at that point the question becomes if we need a transparent interface for opening module sourcefiles or arbitrary files living in packages, that could grab things out of zipfiles (like setuptools has in... one of the modules) -- or other archives of course. (And, yes, I'm zipping up the stdlib for Python 2.7 at Google, to reduce the impact on the aforementioned million of machines :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] peps: PEP 416: remove mentions of mutable/immutable
On 3/10/2012 5:43 AM, victor.stinner wrote: http://hg.python.org/peps/rev/7278026a5db9 changeset: 4124:7278026a5db9 user:Victor Stinner date:Sat Mar 10 11:43:45 2012 +0100 summary: PEP 416: remove mentions of mutable/immutable files: pep-0416.txt | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/pep-0416.txt b/pep-0416.txt --- a/pep-0416.txt +++ b/pep-0416.txt @@ -20,9 +20,8 @@ = A frozendict is a read-only mapping: a key cannot be added nor removed, and a -key is always mapped to the same value. However, frozendict values can be -mutable (not hashable). A frozendict is hashable and so immutable if and only -if all values are hashable (immutable). +key is always mapped to the same value. However, frozendict values can be not +hashable. A frozendict is hashable if and only if all values are hashable. I think this would be better with 'key is always mapped to the same value object'. The revised second sentence does not add anything. Some Python objects are hashable and some not. There is nothing special about frozendict value objects either way. The proper revision of your original is "However, frozendict value may not be hashable." Either change to that or remove it. Terry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Close #14205: dict lookup raises a RuntimeError if the dict is modified during
On Sat, Mar 10, 2012 at 8:27 AM, Victor Stinner wrote: > I ran all these tests, none is still crashing. I don't think that it is > interesting to keep them. Indeed, please add them all back as regular parts of the test suite - this ensures that not only are they fixed now, but they never break again due to other changes. Feel free to create a dedicated "test_fixed_crashers.py" to explain the origins of a fairly eclectic group of tests. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 416: Add a frozendict builtin type
Le 01/03/2012 14:49, Paul Moore a écrit : Just avoid using the term "immutable" at all: You right, I removed mention of mutable/immutable from the PEP. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
