Re: [Python-Dev] Backporting PEP 414
Le 29/02/2012 00:25, Nick Coghlan a écrit : Also, I think there may be some confusion about Armin's plan to handle 3.2 - he aims to write an *import hook* that accepts the u/U prefixes during tokenisation, not a source-to-source transform like 2to3. this needs to be emphasized. Read from the outside, the whole PEP 414 discussion could give the idea that 3.2 is a second class citizen for porting, like 3.0 and 3.1 have been. If such an opinion would spread, that would be bad PR for Python 3 as a whole. Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Spreading the Python 3 religion (was Re: PEP 414 - Unicode Literals for Python 3)
FWIW, I agree that much of the rhetoric in the current version of PEP 414 is excessive. Armin has given me permission to create an updated version of PEP 414 and toning down the hyperbole (or removing it entirely in cases where it's irrelevant to the final decision) is one of the things that I will be changing. I also plan to add a link to Lennart's guide to the various porting strategies that are currently available, more clearly articulate the cases where the new approach can most help (i.e. when there are project specific reasons to avoid the unicode_literals import), as well as name drop Pyramid (Chris McDonough), Flask (Armin), Django (Jacob Kaplan-Moss) and requests (Kenneth Reitz) as cases where key developers of web-related third party frameworks or libraries have indicated that PEP 414 will help greatly with bringing the sections of the Python ecosystem they're involved with into the Python 3 fold over the next few years. My aim is for the end result to better reflect the reasons why Guido *accepted* the PEP, moreso than Armin's own reasons for *wanting* it. Thank you Nick and Armin. I think toning down the rhetoric is a very amicable solution. Let me know if I need to add anything to http://getpython3.com/ (have linked porting guides there too if you want) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
Armin, I see you've (or somebody) changed: As it stands, Python 3 is currently a bad choice for long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. to: As it stands, when chosing between 2.7 and Python 3.2, Python 3 is currently not the best choice for certain long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. Could you just remove the statement completely? Again, my understanding of what is the best choice for certain *long-term* investments is drastically different from yours. In my opinion, python 3 is much more suitable for anything *long-term* than python 2. I don't think that PEPs are the right place to put such polemic and biased statements. Nobody asked you to express your *personal* feelings and thoughts about applicability or state of python3 in the PEP. There are blogs for that. Thank you. - Yury On 2012-02-28, at 11:29 AM, Yury Selivanov wrote: Hi Armin, Could you please remove from the PEP the following statement: As it stands, Python 3 is currently a bad choice for long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. While it may be as such for you, I think it is incorrect for the rest. Moreover, it is harmful for the python 3 adoption, to put such documents on python.org. The python ecosystem is not just limited to WSGI apps, Django and Flask. Yes, we don't have all the packages on pypi support python 3, but many of those are portable within 10 minutes to couple of hours of work (and I did many of such ports for our internal systems.) And many of the essential packages do exist for python 3, like numpy, zeromq etc. I know several sturt-ups, including mine that develop huge commercial applications entirely on python 3. Thanks, -Yury On 2012-02-27, at 5:38 PM, Armin Ronacher wrote: Hi, On 2/27/12 10:18 PM, Terry Reedy wrote: I would like to know if you think that this one change is enough to do agile development and testing, etc, or whether, as Chris McDonough hopes, this is just the first of a series of proposals you have planned. Indeed I have three other PEPs in the work. The reintroduction of except (((ExceptionType),),), the comparision operator and the removal of nonlocal, the latter to make Python 2.x developers feel better about themselves. :-) Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
Armin, I see you've (or somebody) changed: As it stands, Python 3 is currently a bad choice for long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. to: As it stands, when chosing between 2.7 and Python 3.2, Python 3 is currently not the best choice for certain long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. Could you just remove the statement completely? Again, my understanding of what is the best choice for certain *long-term* investments is drastically different from yours. In my opinion, python 3 is much more suitable for anything *long-term* than python 2. I don't think that PEPs are the right place to put such polemic and biased statements. Nobody asked you to express your *personal* feelings and thoughts about applicability or state of python3 in the PEP. There are blogs for that. Thank you. - Yury On 2012-02-28, at 11:29 AM, Yury Selivanov wrote: Hi Armin, Could you please remove from the PEP the following statement: As it stands, Python 3 is currently a bad choice for long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. While it may be as such for you, I think it is incorrect for the rest. Moreover, it is harmful for the python 3 adoption, to put such documents on python.org. The python ecosystem is not just limited to WSGI apps, Django and Flask. Yes, we don't have all the packages on pypi support python 3, but many of those are portable within 10 minutes to couple of hours of work (and I did many of such ports for our internal systems.) And many of the essential packages do exist for python 3, like numpy, zeromq etc. I know several sturt-ups, including mine that develop huge commercial applications entirely on python 3. Thanks, -Yury On 2012-02-27, at 5:38 PM, Armin Ronacher wrote: Hi, On 2/27/12 10:18 PM, Terry Reedy wrote: I would like to know if you think that this one change is enough to do agile development and testing, etc, or whether, as Chris McDonough hopes, this is just the first of a series of proposals you have planned. Indeed I have three other PEPs in the work. The reintroduction of except (((ExceptionType),),), the comparision operator and the removal of nonlocal, the latter to make Python 2.x developers feel better about themselves. :-) Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Feb 29, 2012, at 07:30 AM, Yury Selivanov wrote: As it stands, Python 3 is currently a bad choice for long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. to: As it stands, when chosing between 2.7 and Python 3.2, Python 3 is currently not the best choice for certain long-term investments, since the ecosystem is not yet properly developed, and libraries are still fighting with their API decisions for Python 3. Could you just remove the statement completely? If I read correctly, Nick is undertaking a rewrite of PEP 414, which should help a lot. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 416: Add a frozendict builtin type
As requested, I create a PEP and a related issue: http://www.python.org/dev/peps/pep-0416/ http://bugs.python.org/issue14162 The PEP 416 is different from my previous propositions: frozendict values can be mutable and dict doesn't inherit from frozendict anymore. But it is still possible to use the PyDict C API on frozendict (which is more an implementation detail). TODO: - write the documentation - decide if new functions should be added to the C API, maybe a new PyFrozenDict_New() function? (but would it take a mapping or a list of tuple?) -- PEP: 416 Title: Add a frozendict builtin type Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner victor.stin...@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-February-2012 Python-Version: 3.3 Abstract Add a new frozendict builtin type. Rationale = A frozendict mapping cannot be changed, but its values can be mutable (not hashable). A frozendict is hashable and so immutable if all values are hashable (immutable). Use cases of frozendict: * hashable frozendict can be used as a key of a mapping or as a member of set * frozendict helps optimization because the mapping is constant * frozendict avoids the need of a lock when the frozendict is shared by multiple threads or processes, especially hashable frozendict Constraints === * frozendict has to implement the Mapping abstract base class * frozendict keys and values can be unorderable * a frozendict is hashable if all keys and values are hashable * frozendict hash does not depend on the items creation order Implementation == * Add a PyFrozenDictObject structure based on PyDictObject with an extra Py_hash_t hash; field * frozendict.__hash__() is implemented using hash(frozenset(self.items())) and caches the result in its private hash attribute * Register frozendict has a collections.abc.Mapping * frozendict can be used with PyDict_GetItem(), but PyDict_SetItem() and PyDict_DelItem() raise a TypeError Recipe: immutable dict == An immutable mapping can be implemented using frozendict:: import itertools class immutabledict(frozendict): def __new__(cls, *args, **kw): # ensure that all values are immutable for key, value in itertools.chain(args, kw.items()): if not isinstance(value, (int, float, complex, str, bytes)): hash(value) # frozendict ensures that all keys are immutable return frozendict.__new__(cls, *args, **kw) def __repr__(self): return 'immutabledict' + frozendict.__repr__(self)[10:] Objections == *namedtuple may fit the requiements of a frozendict.* A namedtuple is not a mapping, it does not implement the Mapping abstract base class. *frozendict can be implemented in Python using descriptors and frozendict just need to be practically constant.* If frozendict is used to harden Python (security purpose), it must be implemented in C. A type implemented in C is also faster. *The PEP 351 was rejected.* The PEP 351 tries to freeze an object and so may convert a mutable object to an immutable object (using a different type). frozendict doesn't convert anything: hash(frozendict) raises a TypeError if a value is not hashable. Freezing an object is not the purpose of this PEP. Links = * PEP 412: Key-Sharing Dictionary (`issue #13903 http://bugs.python.org/issue13903`_) * PEP 351: The freeze protocol * `The case for immutable dictionaries; and the central misunderstanding of PEP 351 http://www.cs.toronto.edu/~tijmen/programming/immutableDictionaries.html`_ * `Frozen dictionaries (Python recipe 414283) http://code.activestate.com/recipes/414283-frozen-dictionaries/`_ by Oren Tirosh Copyright = This document has been placed in the public domain. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 416: Add a frozendict builtin type
On Wed, 2012-02-29 at 19:21 +0100, Victor Stinner wrote: As requested, I create a PEP and a related issue: http://www.python.org/dev/peps/pep-0416/ [...snip...] Rationale = A frozendict mapping cannot be changed, but its values can be mutable (not hashable). A frozendict is hashable and so immutable if all values are hashable (immutable). The wording of the above seems very unclear to me. Do you mean A frozendict has a constant set of keys, and for every key, d[key] has a specific value for the lifetime of the frozendict. However, these values *may* be mutable. The frozendict is hashable iff all of the values are hashable. ? (or somesuch) [...snip...] * Register frozendict has a collections.abc.Mapping s/has/as/ ? [...snip...] If frozendict is used to harden Python (security purpose), it must be implemented in C. A type implemented in C is also faster. You mention security purposes here, but this isn't mentioned in the Rationale or Use Cases Hope this is helpful Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 416: Add a frozendict builtin type
Rationale = A frozendict mapping cannot be changed, but its values can be mutable (not hashable). A frozendict is hashable and so immutable if all values are hashable (immutable). The wording of the above seems very unclear to me. Do you mean A frozendict has a constant set of keys, and for every key, d[key] has a specific value for the lifetime of the frozendict. However, these values *may* be mutable. The frozendict is hashable iff all of the values are hashable. ? (or somesuch) [...snip...] I agree that this sentence needs some clarification. David's formulation is also what I would guess it to mean, but it should be stated more explicitly. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Feb 27, 2012, at 10:53 AM, Victor Stinner wrote: A frozendict type is a common request from users and there are various implementations. ISTM, this request is never from someone who has a use case. Instead, it almost always comes from completers, people who see that we have a frozenset type and think the core devs missed the ObviousThingToDo(tm). Frozendicts are trivial to implement, so that is why there are various implementations (i.e. the implementations are more fun to write than they are to use). The frozenset type covers a niche case that is nice-to-have but *rarely* used. Many experienced Python users simply forget that we have a frozenset type. We don't get bug reports or feature requests about the type. When I do Python consulting work, I never see it in a client's codebase. It does occasionally get discussed in questions on StackOverflow but rarely gets offered as an answer (typically on variants of the how do you make a set-of-sets question). If Google's codesearch were still alive, we could add another datapoint showing how infrequently this type is used. I wrote the C implementation for frozensets and the tests that demonstrate their use in problems involving sets-of-sets, yet I have *needed* the frozenset once in my career (for a NFA/DFA conversion algorithm). From this experience, I conclude that adding a frozendict type would be a total waste (except that it would inspire more people to request frozen variante of other containers). Raymond P.S. The one advantage I can see for frozensets and frozendicts is that we have an opportunity to optimize them once they are built (optimizing insertion order to minimize collisions, increasing or decreasing density, eliminating dummy entries, etc). That being said, the same could be accomplished for regular sets and dicts by the addition of an optimize() method. I'm not really enamoured of that idea though because it breaks the abstraction and because people don't seem to need it (i.e. it has never been requested).___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
The frozenset type covers a niche case that is nice-to-have but *rarely* used. Many experienced Python users simply forget that we have a frozenset type. We don't get bug reports or feature requests about the type. When I do Python consulting work, I never see it in a client's codebase. It does occasionally get discussed in questions on StackOverflow but rarely gets offered as an answer (typically on variants of the how do you make a set-of-sets question). If Google's codesearch were still alive, we could add another datapoint showing how infrequently this type is used. snip There are some alternatives to code.google.com, though. For example: http://www.koders.com/default.aspx?s=frozensetsubmit=Searchla=Pythonli=* From a cursory look: quite a bit of the found results are from the various Python implementations, and there is some duplication of projects, but it would be unfair to conclude that frozenset is not being used since many of the results do look legitimate. This is not to argue in favor or against frozendict, just stating that there's still a way to search code online :) Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of PEP-3118 (memoryview part)
Antoine Pitrou solip...@pitrou.net wrote: Stefan Krah ste...@bytereef.org wrote: In Python 3.3 most issues with the memoryview object have been fixed in a recent commit (3f9b3b6f7ff0). Oh and congrats for doing this, of course. Thanks! Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of PEP-3118 (memoryview part)
Greg Ewing greg.ew...@canterbury.ac.nz wrote: Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round. Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work. This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for too complex to backport. I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2-3 or 3-2 porting obstacles. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 411: Provisional packages in the Python standard library
I have updated PEP 411, following the input from this discussion. The updated PEP is at: http://hg.python.org/peps/file/default/pep-0411.txt Changes: - Specified that a package may remain provisional for longer than a single minor release - Shortened the suggested documentation notice, linking to the glossary which will contain the full text - Adding a notice to the package's docstring, which may be programmatically inspected. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On 29 February 2012 19:17, Raymond Hettinger raymond.hettin...@gmail.com wrote: From this experience, I conclude that adding a frozendict type would be a total waste (except that it would inspire more people to request frozen variante of other containers). It would (apparently) help Victor to fix issues in his pysandbox project. I don't know if a secure Python sandbox is an important enough concept to warrant core changes to make it possible. However, if Victor was saying that implementing this PEP was all that is needed to implement a secure sandbox, then that would be a very different claim, and likely much more compelling (to some, at least - I have no personal need for a secure sandbox). Victor quotes 6 implementations. I don't see any rationale (either in the email that started this thread, or in the PEP) to explain why these aren't good enough, and in particular why the implementation has to be in the core. There's the hint in the PEP If frozendict is used to harden Python (security purpose), it must be implemented in C. But why in the core (as opposed to an extension)? And why and how would frozendict help in hardening Python? As it stands, I don't find the PEP compelling. The hardening use case might be significant but Victor needs to spell it out if it's to make a difference. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Thu, Mar 1, 2012 at 7:08 AM, Paul Moore p.f.mo...@gmail.com wrote: As it stands, I don't find the PEP compelling. The hardening use case might be significant but Victor needs to spell it out if it's to make a difference. +1 Avoiding-usenet-nod-syndrome'ly, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
On Feb 28, 2012 7:14 PM, mar...@v.loewis.de wrote: Why is readding u'' a feature and not a bug? There is a really simple litmus test for whether something is a bug: does it deviate from the specification? In this case, the specification is the grammar, and the implementation certainly doesn't deviate from it. So it can't be a bug. I don't think anyone can assert that the specification itself is immune to having bugs. Regards, Martin P.S. Before anybody over-interprets this criterion: there is certain implicit behavior assumed in Python that may not actually be documented, such as the interpreter will not core dump, and the source code will compile with any standard C compiler. Deviation from these implicit assumption is also a bug. However, they don't apply here. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
On Wed, 29 Feb 2012 17:06:21 -0500, Calvin Spealman ironfro...@gmail.com wrote: On Feb 28, 2012 7:14 PM, mar...@v.loewis.de wrote: Why is readding u'' a feature and not a bug? There is a really simple litmus test for whether something is a bug: does it deviate from the specification? In this case, the specification is the grammar, and the implementation certainly doesn't deviate from it. So it can't be a bug. I don't think anyone can assert that the specification itself is immune to having bugs. Yes, but specification bug fixes are new features :) --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
There is a really simple litmus test for whether something is a bug: does it deviate from the specification? In this case, the specification is the grammar, and the implementation certainly doesn't deviate from it. So it can't be a bug. I don't think anyone can assert that the specification itself is immune to having bugs. I can assert that - a specification inherently cannot be incorrect. It can only be unintentional. There are certainly documentation errors. They occur when the documentation deviates from the implementation *and* from the intent. They are easy to fix in a bug fix release (assuming the implementation correctly reflects the intent). But then, this isn't the case here, either: the *intent* of the current grammar is that there is no u prefix in the Python 3 language. So the specification clearly corresponds to the intent also. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Thu, Mar 1, 2012 at 8:08 AM, Paul Moore p.f.mo...@gmail.com wrote: It would (apparently) help Victor to fix issues in his pysandbox project. I don't know if a secure Python sandbox is an important enough concept to warrant core changes to make it possible. If a secure Python sandbox had been available last year, we would probably be still using Python at work for end-user scripting, instead of having had to switch to Javascript. At least, that would be the case if this sandbox is what I think it is (we embed a scripting language in our C++ main engine, and allow end users to customize and partly drive our code). But features enabling that needn't be core; I wouldn't object to having to get some third-party add-ons to make it all work. Chris Angelico ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Feb 29, 2012, at 1:08 PM, Paul Moore wrote: As it stands, I don't find the PEP compelling. The hardening use case might be significant but Victor needs to spell it out if it's to make a difference. If his sandboxing project needs it, the type need not be public. It can join dictproxy and structseq in our toolkit of internal types. Adding frozendict() as a new public type is unnecessary and undesirable -- a proliferation of types makes it harder to decide which tool is the most appropriate for a given problem. The itertools module ran into the issue early. Adding a new itertool tends to make the whole module harder to figure-out. Raymond P.S ISTM that lately Python is growing fatter without growing more powerful or expressive. Generators, context managers, and decorators were honking good ideas -- we need more of those rather than minor variations on things we already have. Plz forgive the typos -- I'm typing with one hand -- the other is holding a squiggling baby :-)___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
It would (apparently) help Victor to fix issues in his pysandbox project. I don't know if a secure Python sandbox is an important enough concept to warrant core changes to make it possible. Ok, let's talk about sandboxing and security. The main idea of pysandbox is to reuse most of CPython but hide dangerous functions and run untrusted code in a separated namespace. The problem is to create the sandbox and ensure that it is not possible to escape from this sandbox. pysandbox is still a proof-of-concept, even if it works pretty well for short dummy scripts. But pysandbox is not ready for real world programs. pysandbox uses various tricks and hacks to create a sandbox. But there is a major issue: the __builtins__ dict (or module) is available and used everywhere (in module globals, in frames, in functions globals, etc.), and can be modified. A read-only __builtins__ dict is required to protect the sandbox. If the untrusted can modify __builtins__, it can replace core functions like isinstance(), len(), ... and so modify code outside the sandbox. To implement a frozendict in Python, pysandbox uses the blacklist approach: a class inheriting from dict and override some methods to raise an error. The whitelist approach cannot be used for a type implemented in Python, because the __builtins__ type must inherit from dict: ceval.c expects a type compatible with PyDict_GetItem and PyDict_SetItem. Problem: if you implement a frozendict type inheriting from dict in Python, it is still possible to call dict methods (e.g. dict.__setitem__()). To fix this issue, pysandbox removes all dict methods modifying the dict: __setitem__, __delitem__, pop, etc. This is a problem because untrusted code cannot use these methods on valid dict created in the sandbox. However, if Victor was saying that implementing this PEP was all that is needed to implement a secure sandbox, then that would be a very different claim, and likely much more compelling (to some, at least - I have no personal need for a secure sandbox). A builtin frozendict type compatible with the PyDict C API is very convinient for pysandbox because using this type for core features like builtins requires very few modification. For example, use frozendict for __builtins__ only requires to modify 3 lines in frameobject.c. I don't see how to solve the pysandbox issue (read-only __builtins__ issue, need to remove dict.__setitem__ friends) without modifying CPython (so without adding a frozendict type). As it stands, I don't find the PEP compelling. The hardening use case might be significant but Victor needs to spell it out if it's to make a difference. I don't know if hardening Python is a compelling argument to add a new builtin type. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Thu, 01 Mar 2012 10:13:01 +1100, Chris Angelico ros...@gmail.com wrote: On Thu, Mar 1, 2012 at 8:08 AM, Paul Moore p.f.mo...@gmail.com wrote: It would (apparently) help Victor to fix issues in his pysandbox project. I don't know if a secure Python sandbox is an important enough concept to warrant core changes to make it possible. If a secure Python sandbox had been available last year, we would probably be still using Python at work for end-user scripting, instead of having had to switch to Javascript. At least, that would be the case if this sandbox is what I think it is (we embed a scripting language in our C++ main engine, and allow end users to customize and partly drive our code). But features enabling that needn't be core; I wouldn't object to having to get some third-party add-ons to make it all work. I likewise am aware of a project where the availability of sandboxing might be make-or-break for continuing to use Python. In this case the idea would be sandboxing plugins called from a Python main program. I *think* that Victor's project would enable that, but I haven't looked at it closely. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Feb 29, 2012, at 3:52 PM, Victor Stinner wrote: I don't know if hardening Python is a compelling argument to add a new builtin type. It isn't. Builtins are for general purpose use. It is not something most people should use; however, if it is a builtin, people will be drawn to frozendicts like moths to a flame. The tuple-as-frozenlist anti-pattern shows what we're up against. Another thought: if pypy is successful at providing sandboxing, the need for sandboxing in CPython is substantially abated. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
A frozendict type is a common request from users and there are various implementations. ISTM, this request is never from someone who has a use case. One of my colleagues implemented recently its own frozendict class (which the frozendict name ;-)). He tries to implement something like the PEP 351, not a generic freeze() function but a specialized function for his use case (only support list/tuple and dict/frozendict if I remember correctly). It remembers me the question: why does Python not provide a frozendict type? Even if it is not possible to write a perfect freeze() function, it looks like some developers need sort of this function and I hope that frozendict would be a first step in the good direction. Ruby has a freeze method. On a dict, it provides the same behaviour than frozendict: the mapping cannot be modified anymore, but values are still mutable. http://ruby-doc.org/core-1.9.3/Object.html#method-i-freeze Many experienced Python users simply forget that we have a frozenset type. We don't get bug reports or feature requests about the type. I used it in my previous work to declare the access control list (ACL) on services provided by XML-RPC object. To be honest, set could also be used, but I chose frozenset to ensure that my colleagues don't try to modify it without understanding the consequences of such change. It was not a protecting against evil hackers from the Internet, but from my colleagues :-) Sorry, I didn't find any bug in frozenset :-) My usage was just to declare a frozendict and then check if an item is in the set, and it works pretty well! P.S. The one advantage I can see for frozensets and frozendicts is that we have an opportunity to optimize them once they are built (optimizing insertion order to minimize collisions, increasing or decreasing density, eliminating dummy entries, etc). That being said, the same could be accomplished for regular sets and dicts by the addition of an optimize() method. You can also implement more optimizations in Python peephole or PyPy JIT because the mapping is constant and so you can do the lookup at compilation, instead of doing it at runtime. Dummy example: --- config = frozendict(debug=False) if config['debug']: enable_debug() --- config['debug'] is always False and so you can just drop the call to enable_debug() while compiling this code. It would avoid the need of a preprocessor in some cases (especially conditional code, like the C #ifdef). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
Raymond Hettinger wrote: On Feb 29, 2012, at 3:52 PM, Victor Stinner wrote: I don't know if hardening Python is a compelling argument to add a new builtin type. It isn't. Builtins are for general purpose use. It is not something most people should use; however, if it is a builtin, people will be drawn to frozendicts like moths to a flame. The tuple-as-frozenlist anti-pattern shows what we're up against. Perhaps I'm a little slow today, but I don't get this. Could you elaborate on tuple-as-frozenlist anti-pattern please? i.e. what it is, why it is an anti-pattern, and examples of it in real life? -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of PEP-3118 (memoryview part)
On 2/29/2012 2:34 PM, Stefan Krah wrote: Greg Ewinggreg.ew...@canterbury.ac.nz wrote: Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round. Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work. This *almost* sounds like a feature addition. This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for too complex to backport. I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2-3 or 3-2 porting obstacles. Stefan Krah -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
Armin filed and argued for the addition in a PEP, a Python *Enhancement* Proposal. He did not file a bugfix behavior issue on the tracker. Let us leave it as that. x.y is a specified language. We continuously improve the x.y docs that describe and explain the specification. We also improve the implementation of x.y and periodically release the improvements as x.y.z. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
Raymond Hettinger wrote: On Feb 27, 2012, at 10:53 AM, Victor Stinner wrote: A frozendict type is a common request from users and there are various implementations. ISTM, this request is never from someone who has a use case. Instead, it almost always comes from completers, people who see that we have a frozenset type and think the core devs missed the ObviousThingToDo(tm). Frozendicts are trivial to implement, so that is why there are various implementations (i.e. the implementations are more fun to write than they are to use). They might be trivial for *you*, but the fact that people keep asking for help writing a frozendict, or stating that their implementation sucks, demonstrates that for the average Python coder they are not trivial at all. And the implementations I've seen don't seem to be so much fun as *tedious*. E.g. google on python frozendict and the second link is from somebody who had tried for a couple of days and is still not happy: http://python.6.n6.nabble.com/frozendict-td4377791.html You may dismiss him as a completer, but what is asserted without evidence can be rejected without evidence, and so we may just as well declare that he has a brilliantly compelling use-case, if only we knew what it was... wink I see one implementation on ActiveState that has at least one serious problem, reported by you: http://code.activestate.com/recipes/414283-frozen-dictionaries/ So I don't think we can dismiss frozendict as trivial. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Feb 29, 2012, at 4:23 PM, Victor Stinner wrote: One of my colleagues implemented recently its own frozendict class (which the frozendict name ;-) I write new collection classes all the time. That doesn't mean they warrant inclusion in the library or builtins. There is a use case for ListenableSets and ListenableDicts -- do we need them in the library? I think not. How about case insensitive variants? I think not. There are tons of recipes on ASPN and on PyPI. That doesn't make them worth adding in to the core group of types. As core developers, we need to place some value on language compactness and learnability. The language has already gotten unnecessarily fat -- it is the rare Python programmer who knows set operations on dict views, new-style formatting, abstract base classes, contextlib/functools/itertools, how the with-statement works, how super() works, what properties/staticmethods/classmethods are for, differences between new and old-style classes, Exception versus BaseException, weakreferences, __slots__, chained exceptions, etc. If we were to add another collections type, it would need to be something that powerfully adds to the expressivity of the language. Minor variants on what we already have just makes that language harder to learn and remember but not providing much of a payoff in return. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of PEP-3118 (memoryview part)
[erroneouly hit send button before instead of edit menu above it] On 2/29/2012 2:34 PM, Stefan Krah wrote: Greg Ewinggreg.ew...@canterbury.ac.nz wrote: Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round. If implementation and PEP conflict, the normal question is 'what does the doc say?' as doc takes precedent over PEP. However, in this case the 'MemoryView objects' section under 'Concrete objects' says nothing about the above that I could see and refers to Buffer Protocal in Abstract Objects Layer. I did not see anything there either, but could have missed it. Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work. This looks a bit like an enhancement ;-) This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for too complex to backport. My comment was more 'unnecessary to backpart'. This is based on the following thoughts (which could have mistakes). * I do not see enough benefit that I could wish you to write or anyone else to review a bugfix patch. I would in no way stop you if this continue to itch you ;-). * Sorting out bugfix changes from feature looks complex and possibly contentious and might take some time to discuss. * 3.2.3 is, I presume, less than a month away, and if that is missed, the next and last bugfix will be 3.2.4 at about the same time as 3.3.0. At that time, the full new memoryview version would be a better target. * As for porting, my impression is that the PEP directly affects only C code and Python code using ctypes and only some fraction of those. If the bugfix-only patch is significantly different from complete patch, porting to 3.2 would be significantly different from porting to 3.3. So I can foresee a temptation to just port to 3.3 anyway. I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2-3 or 3-2 porting obstacles. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] State of PEP-3118 (memoryview part)
On Thu, Mar 1, 2012 at 11:48 AM, Terry Reedy tjre...@udel.edu wrote: * As for porting, my impression is that the PEP directly affects only C code and Python code using ctypes and only some fraction of those. If the bugfix-only patch is significantly different from complete patch, porting to 3.2 would be significantly different from porting to 3.3. So I can foresee a temptation to just port to 3.3 anyway. memoryview as it exists in 2.7 and 3.2 misbehaves when used with certain buffer exporters - while Antoine bashed it into shape (mostly) for 1D views into 1D objects, it's rather temperamental if you try to go beyond that. So it affects the Python level as well, in terms of what objects are likely to upset memoryview. Still, I think backporting would be a lot of work for relatively small benefit, so it ends up in my with infinite resources, sure, but with limited resources, there are more fruitful things to be doing pile. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On Wed, Feb 29, 2012 at 3:52 PM, Victor Stinner victor.stin...@haypocalc.com wrote: It would (apparently) help Victor to fix issues in his pysandbox project. I don't know if a secure Python sandbox is an important enough concept to warrant core changes to make it possible. Ok, let's talk about sandboxing and security. The main idea of pysandbox is to reuse most of CPython but hide dangerous functions and run untrusted code in a separated namespace. The problem is to create the sandbox and ensure that it is not possible to escape from this sandbox. pysandbox is still a proof-of-concept, even if it works pretty well for short dummy scripts. But pysandbox is not ready for real world programs. I hope you have studied (recent) history. Sandboxes in Python traditionally have not been secure. Read the archives for details. pysandbox uses various tricks and hacks to create a sandbox. But there is a major issue: the __builtins__ dict (or module) is available and used everywhere (in module globals, in frames, in functions globals, etc.), and can be modified. A read-only __builtins__ dict is required to protect the sandbox. If the untrusted can modify __builtins__, it can replace core functions like isinstance(), len(), ... and so modify code outside the sandbox. To implement a frozendict in Python, pysandbox uses the blacklist approach: a class inheriting from dict and override some methods to raise an error. The whitelist approach cannot be used for a type implemented in Python, because the __builtins__ type must inherit from dict: ceval.c expects a type compatible with PyDict_GetItem and PyDict_SetItem. Problem: if you implement a frozendict type inheriting from dict in Python, it is still possible to call dict methods (e.g. dict.__setitem__()). To fix this issue, pysandbox removes all dict methods modifying the dict: __setitem__, __delitem__, pop, etc. This is a problem because untrusted code cannot use these methods on valid dict created in the sandbox. However, if Victor was saying that implementing this PEP was all that is needed to implement a secure sandbox, then that would be a very different claim, and likely much more compelling (to some, at least - I have no personal need for a secure sandbox). A builtin frozendict type compatible with the PyDict C API is very convinient for pysandbox because using this type for core features like builtins requires very few modification. For example, use frozendict for __builtins__ only requires to modify 3 lines in frameobject.c. I don't see how to solve the pysandbox issue (read-only __builtins__ issue, need to remove dict.__setitem__ friends) without modifying CPython (so without adding a frozendict type). As it stands, I don't find the PEP compelling. The hardening use case might be significant but Victor needs to spell it out if it's to make a difference. I don't know if hardening Python is a compelling argument to add a new builtin type. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 416: Add a frozendict builtin type
In http://mail.python.org/pipermail/python-dev/2012-February/117113.html Victor Stinner posted: An immutable mapping can be implemented using frozendict:: class immutabledict(frozendict): def __new__(cls, *args, **kw): # ensure that all values are immutable for key, value in itertools.chain(args, kw.items()): if not isinstance(value, (int, float, complex, str, bytes)): hash(value) # frozendict ensures that all keys are immutable return frozendict.__new__(cls, *args, **kw) What is the purpose of this? Is it just a hashable frozendict? If it is for security (as some other messages suggest), then I don't think it really helps. class Proxy: def __eq__(self, other): return self.value == other def __hash__(self): return hash(self.value) An instance of Proxy is hashable, and the hash is not object.hash, but it is still mutable. You're welcome to call that buggy, but a secure sandbox will have to deal with much worse. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP czar for PEP 3144?
Just checking in: On Mon, Feb 20, 2012 at 5:48 PM, Nick Coghlan ncogh...@gmail.com wrote: At the very least: - the IP Interface API needs to move to a point where it more clearly *is* an IP Address and *has* an associated IP Network (rather than being the other way around) This is done [1]. There's cleanup that needs to happen here, but the interface classes are now subclasses of the respective address classes. Now I need to apply some consistency and then move on to the remaining issues points: - IP Network needs to behave more like an ordered set of sequential IP Addresses (without sometimes behaving like an Address in its own right) - iterable APIs should consistently produce iterators (leaving users free to wrap list() around the calls if they want the concrete realisation) Cheers, peter [1] http://code.google.com/p/ipaddress-py/source/detail?r=10dd6a68139fb99116219865afcd1c183777e8cc (the date is munged b/c I rebased to my original commit before submitting). -- Peter Moody Google 1.650.253.7306 Security Engineer pgp:0xC3410038 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
On 01.03.2012 02:45, Raymond Hettinger wrote: On Feb 29, 2012, at 4:23 PM, Victor Stinner wrote: One of my colleagues implemented recently its own frozendict class (which the frozendict name ;-) I write new collection classes all the time. That doesn't mean they warrant inclusion in the library or builtins. There is a use case for ListenableSets and ListenableDicts -- do we need them in the library? I think not. How about case insensitive variants? I think not. There are tons of recipes on ASPN and on PyPI. That doesn't make them worth adding in to the core group of types. +1. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP czar for PEP 3144?
On Thu, Mar 1, 2012 at 3:13 PM, Peter Moody pmo...@google.com wrote: Just checking in: On Mon, Feb 20, 2012 at 5:48 PM, Nick Coghlan ncogh...@gmail.com wrote: At the very least: - the IP Interface API needs to move to a point where it more clearly *is* an IP Address and *has* an associated IP Network (rather than being the other way around) This is done [1]. There's cleanup that needs to happen here, but the interface classes are now subclasses of the respective address classes. Thanks for the update! I'll be moving house this month, which may disrupt things a bit, but I'll still be trying to keep up with email, etc. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a frozendict builtin type
01.03.12 01:52, Victor Stinner написав(ла): Problem: if you implement a frozendict type inheriting from dict in Python, it is still possible to call dict methods (e.g. dict.__setitem__()). To fix this issue, pysandbox removes all dict methods modifying the dict: __setitem__, __delitem__, pop, etc. This is a problem because untrusted code cannot use these methods on valid dict created in the sandbox. You can redefine dict.__setitem__. oldsetitem = dict.__setitem__ def newsetitem(self, value): # check if self is not frozendict ... oldsetitem(self, value) dict.__setitem__ = newsetitem ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
