Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

>> Problem: if you implement a frozendict type inheriting from dict in
>> Python, it is still possible to call dict methods (e.g.
>> dict.__setitem__()). To fix this issue, pysandbox removes all dict
>> methods modifying the dict: __setitem__, __delitem__, pop, etc. This
>> is a problem because untrusted code cannot use these methods on valid
>> dict created in the sandbox.
>
>
> You can redefine dict.__setitem__.

Ah? It doesn't work here.

>>> dict.__setitem__=lambda key, value: None
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can't set attributes of built-in/extension type 'dict'

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Giampaolo Rodolà]

Il 01 marzo 2012 02:45, Raymond Hettinger
 ha scritto:
>
> On Feb 29, 2012, at 4:23 PM, Victor Stinner wrote:
>
> One of my colleagues implemented recently its own frozendict class
> (which the "frozendict" name ;-)
>
>
> I write new collection classes all the time.
> That doesn't mean they warrant inclusion in the library or builtins.
> There is a use case for ListenableSets and ListenableDicts -- do we
> need them in the library?  I think not.  How about case insensitive
> variants?
> I think not.  There are tons of recipes on ASPN and on PyPI.
> That doesn't make them worth adding in to the core group of types.
>
> As core developers, we need to place some value on language
> compactness and learnability.  The language has already gotten
> unnecessarily fat -- it is the rare Python programmer who knows
> set operations on dict views, new-style formatting, abstract base classes,
> contextlib/functools/itertools, how the with-statement works,
> how super() works, what properties/staticmethods/classmethods are for,
> differences between new and old-style classes, Exception versus
> BaseException,
> weakreferences, __slots__, chained exceptions, etc.
>
> If we were to add another collections type, it would need to be something
> that powerfully adds to the expressivity of the language.  Minor variants
> on what we already have just makes that language harder to learn and
> remember
> but not providing much of a payoff in return.
>
>
> Raymond
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>

+1

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

On Wednesday 29 February 2012 20:17:05 Raymond Hettinger wrote:
> On Feb 27, 2012, at 10:53 AM, Victor Stinner wrote:
> > A frozendict type is a common request from users and there are various
> > implementations.
>
> ISTM, this request is never from someone who has a use case.
> Instead, it almost always comes from "completers", people
> who see that we have a frozenset type and think the core devs
> missed the ObviousThingToDo(tm).  Frozendicts are trivial to
> implement, so that is why there are various implementations
> (i.e. the implementations are more fun to write than they are to use).
>
> The frozenset type covers a niche case that is nice-to-have but
> *rarely* used.  Many experienced Python users simply forget
> that we have a frozenset type.  We don't get bug reports or
> feature requests about the type.  When I do Python consulting
> work, I never see it in a client's codebase.  It does occasionally
> get discussed in questions on StackOverflow but rarely gets
> offered as an answer (typically on variants of the "how do you
> make a set-of-sets" question).  If Google's codesearch were still
> alive, we could add another datapoint showing how infrequently
> this type is used.


Here are my real-world use cases. Not for security, but for safety and 
performance reasons (I've built by own RODict and ROList modeled after 
dictproxy):

- Global, but immutable containers, e.g. as class members

- Caching. My data container objects (say, resultsets from a db or something) 
  usually inherit from list or dict (sometimes also set) and are cached 
  heavily. In order to ensure that they are not modified (accidentially), I 
  have to choices: deepcopy or immutability. deepcopy is so expensive, that 
  it's often cheaper to just leave out the cache. So I use immutability. (oh 
  well, the objects are further restricted with __slots__)

I agree, these are not general purpose issues, but they are not *rare*, I'd 
think.

nd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

>> The main idea of pysandbox is to reuse most of CPython but hide
>> "dangerous" functions and run untrusted code in a separated namespace.
>> The problem is to create the sandbox and ensure that it is not
>> possible to escape from this sandbox. pysandbox is still a
>> proof-of-concept, even if it works pretty well for short dummy
>> scripts. But pysandbox is not ready for real world programs.
>
> I hope you have studied (recent) history. Sandboxes in Python
> traditionally have not been secure. Read the archives for details.

The design of pysandbox makes it difficult to implement. It is mostly
based on blacklist, so any omission would lead to a vulnerability. I
read the recent history of sandboxes and see other security modules
for Python, and I don't understand your reference to  "Sandboxes in
Python traditionally have not been secure." There is no known
vulnerability in pysandbox, did I miss something? (there is only a
limitation on the dict API because of the lack of frozendict.)

Are you talking about rexec/Bastion? (which cannot be qualified as "recent" :-))

pysandbox limitations are documented in its README file:

<< pysandbox is a sandbox for the Python namespace, not a sandbox between Python
and the operating system. It doesn't protect your system against Python
security vulnerabilities: vulnerabilities in modules/functions available in
your sandbox (depend on your sandbox configuration). By default, only few
functions are exposed to the sandbox namespace which limits the attack surface.

pysandbox is unable to limit the memory of the sandbox process: you have to use
your own protection. >>

Hum, I am also not sure that pysandbox "works" with threads :-) I mean
that enabling pysandbox impacts all running threads, not only one
thread, which can cause issues. It should also be mentioned.

PyPy sandbox has a different design: it uses a process with no
priviledge, all syscalls are redirected to another process which apply
security checks to each syscall.
http://doc.pypy.org/en/latest/sandbox.html

See also the seccomp-nurse project, a generic sandbox using Linux SECCOMP:
http://chdir.org/~nico/seccomp-nurse/

See also pysandbox README for a list of other Python security modules.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] State of PEP-3118 (memoryview part)

[Stefan Krah]

Terry Reedy  wrote:
 Options 2) and 3) would ideally entail one backwards
 incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview
 with format 'B' rejects integers but accepts byte objects, but
 according to the struct syntax mandated by the PEP it should be
 the other way round.
>
> If implementation and PEP conflict, the normal question is 'what does  
> the doc say?' as doc takes precedent over PEP. However, in this case the  
> 'MemoryView objects' section under 'Concrete objects' says nothing about  
> the above that I could see and refers to Buffer Protocal in Abstract  
> Objects Layer. I did not see anything there either, but could have  
> missed it.

For the C-API, it's here:

http://docs.python.org/py3k/c-api/buffer.html#the-buffer-structure

const char *format

  A NULL terminated string in struct module style syntax giving the
  contents of the elements available through the buffer. If this is NULL,
  "B" (unsigned bytes) is assumed.


Unfortunately, the memoryview documentation itself gives examples
like these:

http://docs.python.org/py3k/library/stdtypes.html#typememoryview

data = bytearray(b'abcefg')
v = memoryview(data)
v[0] = b'z' # That's where the struct module would throw an error.


> * 3.2.3 is, I presume, less than a month away, and if that is missed,  
> the next and last bugfix will be 3.2.4 at about the same time as 3.3.0.  

That would be too soon indeed.


> * As for porting, my impression is that the PEP directly affects only C
> code and Python code using ctypes and only some fraction of those. If
> the bugfix-only patch is significantly different from complete patch,
> porting to 3.2 would be significantly different from porting to 3.3. So
> I can foresee a temptation to just port to 3.3 anyway.

The general problem is this: If someone supports 2 and 3 and uses the
single codebase approach, it's unlikely that new features will ever be
used. Even if a new 3.3 project is started that needs to be backwards
compatible, I can imagine that people will shun anything that 3to2 isn't
able to handle (out of the box).


This would be less of an issue if the officially sanctioned way of porting
were the "separate branches with 2to3 (or 3to2) for the initial conversion"
approach.


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Stefan Krah]

Brett Cannon  wrote:
> Changes to http://docs.python.org/howto/pyporting.html are welcome. I tried to
> make sure it exposed all possibilities with tips on how to support as far back
> as Python 2.5.

I'd like to add a section that highlights the advantages of separate
branches. Starting perhaps with:

Advantages of separate branches:

  1) The two code bases are cleaner.

  2) Neither version is a second class citizen.

  3) New Python3 features can be adopted without worrying about conversion 
tools.

  4) For the developer: psychologically, slowly the py3k version becomes the
 master branch (as it should).

  5) For the user: running 2to3 on install sends the signal that version 2
 is the real version. This is not the case if there are, say, src2/
 and src3/ directories in the distribution.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 416: Add a frozendict builtin type

[Victor Stinner]

>> Rationale
>> =
>>
>> A frozendict mapping cannot be changed, but its values can be mutable
>> (not hashable). A frozendict is hashable and so immutable if all
>> values are hashable (immutable).
> The wording of the above seems very unclear to me.
>
> Do you mean "A frozendict has a constant set of keys, and for every key,
> d[key] has a specific value for the lifetime of the frozendict.
> However, these values *may* be mutable.  The frozendict is hashable iff
> all of the values are hashable." ?  (or somesuch)

New try:

"A frozendict is a read-only mapping: a key cannot be added nor
removed, and a key is always mapped to the same value. However,
frozendict values can be mutable (not hashable). A frozendict is
hashable and so immutable if and only if all values are hashable
(immutable)."

>>  * Register frozendict has a collections.abc.Mapping
> s/has/as/ ?

Oops, fixed.

>> If frozendict is used to harden Python (security purpose), it must be
>> implemented in C. A type implemented in C is also faster.
>
> You mention security purposes here, but this isn't mentioned in the
> Rationale or Use Cases

I added two use cases: security sandbox and cache.

> Hope this is helpful

Yes, thanks.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Merlijn van Deen]

On 1 March 2012 12:11, Stefan Krah  wrote:

> Advantages of separate branches:
>

Even though I agree on most of your points, I disagree with

 2) Neither version is a second class citizen.


In my experience, this is only true if you have a very strict discipline,
or if both branches are used a lot. If there are two branches (say: py2 and
py3), and one is used much less (say: py3), that one will always be the
second class citizen - the py2 branch, which is used by 'most people' gets
more feature requests and bug reports. People will implement the features
and bug fixes in the py2 branch, and sometimes forget to port them to the
py3 branch, which means the branches start diverging. This divergence makes
applying newer changes even more difficult, leading to further divergence.

Another cause for this is the painful merging in most version control
systems. I'm guessing you all know the pain of 'svn merge' - and there are
a *lot *of projects still using SVN or even CVS.

As such, you need to impose the discipline to always apply changes to both
branches. This is a reasonable thing for larger projects, but it is
generally harder to implement it for smaller projects, as you're already
lucky people are actually contributing.

Best,
Merlijn
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 416: Add a frozendict builtin type

[Victor Stinner]

>> An immutable mapping can be implemented using frozendict::
>
>>     class immutabledict(frozendict):
>>         def __new__(cls, *args, **kw):
>>             # ensure that all values are immutable
>>             for key, value in itertools.chain(args, kw.items()):
>>                 if not isinstance(value, (int, float, complex, str, bytes)):
>>                     hash(value)
>>             # frozendict ensures that all keys are immutable
>>             return frozendict.__new__(cls, *args, **kw)
>
> What is the purpose of this?  Is it just a hashable frozendict?

It's an hashable frozendict or a "really frozen dict" or just "an
immutable dict". It helps to detect errors earlier when you need a
hashable frozendict. It is faster than hash(frozendict) because it
avoids to hash known immutable types.

If the recipe is confusion, it can be removed. Or it may be added to
collections or somewhere else.

> If it is for security (as some other messages suggest), then I don't
> think it really helps.
>
>    class Proxy:
>        def __eq__(self, other): return self.value == other
>        def __hash__(self): return hash(self.value)
>
> An instance of Proxy is hashable, and the hash is not object.hash,
> but it is still mutable.  You're welcome to call that buggy, but a
> secure sandbox will have to deal with much worse.

Your example looks to be incomplete: where does value come from? Is it
supposed to be a read-only view of an object?

Such Proxy class doesn't help to implement a sandbox because
Proxy.value can be modified. I use closures to implement proxies in
pysandbox. Dummy example:

def createLengthProxy(secret):
  class Proxy:
def __len__(self):
  return len(secret)
  return Proxy()

Such proxy is not safe because it is possible to retrieve the secret:

secret = "abc"
value = createLengthProxy(secret).__len__.__closure__[0].cell_contents
assert value is secret

pysandbox implements other protections to block access to __closure__.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 416: Add a frozendict builtin type

[Victor Stinner]

>> Rationale
>> =
>>
>> A frozendict mapping cannot be changed, but its values can be mutable
>> (not hashable). A frozendict is hashable and so immutable if all
>> values are hashable (immutable).
> The wording of the above seems very unclear to me.
>
> Do you mean "A frozendict has a constant set of keys, and for every key,
> d[key] has a specific value for the lifetime of the frozendict.
> However, these values *may* be mutable.  The frozendict is hashable iff
> all of the values are hashable." ?  (or somesuch)

New try:

"A frozendict is a read-only mapping: a key cannot be added nor
removed, and a key is always mapped to the same value. However,
frozendict values can be mutable (not hashable). A frozendict is
hashable and so immutable if and only if all values are hashable
(immutable)."

>>  * Register frozendict has a collections.abc.Mapping
> s/has/as/ ?

Oops, fixed.

>> If frozendict is used to harden Python (security purpose), it must be
>> implemented in C. A type implemented in C is also faster.
>
> You mention security purposes here, but this isn't mentioned in the
> Rationale or Use Cases

I added two use cases: security sandbox and cache.

> Hope this is helpful

Yes, thanks.


Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Yury Selivanov]

Actually I find fronzendict concept quite useful.  We also have an
implementation in our framework, and we use it, for instance, in
http request object, for parsed arguments and parsed forms, which
values shouldn't be ever modified once parsed.

Of course everybody can live without it, but given the fact of how
easy it is to implement it I think its OK to have it.

+1.

On 2012-02-29, at 8:45 PM, Raymond Hettinger wrote:

> 
> On Feb 29, 2012, at 4:23 PM, Victor Stinner wrote:
> 
>> One of my colleagues implemented recently its own frozendict class
>> (which the "frozendict" name ;-)
> 
> I write new collection classes all the time.
> That doesn't mean they warrant inclusion in the library or builtins.
> There is a use case for ListenableSets and ListenableDicts -- do we
> need them in the library?  I think not.  How about case insensitive variants?
> I think not.  There are tons of recipes on ASPN and on PyPI.  
> That doesn't make them worth adding in to the core group of types.
> 
> As core developers, we need to place some value on language 
> compactness and learnability.  The language has already gotten
> unnecessarily fat -- it is the rare Python programmer who knows
> set operations on dict views, new-style formatting, abstract base classes,
> contextlib/functools/itertools, how the with-statement works, 
> how super() works, what properties/staticmethods/classmethods are for,
> differences between new and old-style classes, Exception versus BaseException,
> weakreferences, __slots__, chained exceptions, etc.
> 
> If we were to add another collections type, it would need to be something
> that powerfully adds to the expressivity of the language.  Minor variants 
> on what we already have just makes that language harder to learn and remember
> but not providing much of a payoff in return.
> 
> 
> Raymond
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Lennart Regebro]

I also don't agree with the claim that a py3 version using 2to3 is a
"second class citizen". You need to adopt the Python 2 code to Python
3 in that case too, and none of the overrules the other.

//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

> A builtin frozendict type "compatible" with the PyDict C API is very
> convinient for pysandbox because using this type for core features
> like builtins requires very few modification. For example, use
> frozendict for __builtins__ only requires to modify 3 lines in
> frameobject.c.

See the frozendict_builtins.patch attached to the issue #14162. Last version:
http://bugs.python.org/file24690/frozendict_builtins.patch

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

> Here are my real-world use cases. Not for security, but for safety and
> performance reasons (I've built by own RODict and ROList modeled after
> dictproxy):
>
> - Global, but immutable containers, e.g. as class members

I attached type_final.patch to the issue #14162 to demonstrate how
frozendict can be used to implement a "read-only" type. Last version:
http://bugs.python.org/file24696/type_final.patch

Example:

>>> class FinalizedType:
...   __final__=True
...   attr = 10
...   def hello(self):
... print("hello")
...
>>> FinalizedType.attr=12
TypeError: 'frozendict' object does not support item assignment
>>> FinalizedType.hello=print
TypeError: 'frozendict' object does not support item assignment

(instance do still have a mutable dict)

My patch checks for the __final__ class attribute, but the conversion
from dict to frozendict may be done by a function or a type method.
Creating a read-only type is a different issue, it's just another
example of frozendict usage.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

On Thursday 01 March 2012 14:07:10 Victor Stinner wrote:
> > Here are my real-world use cases. Not for security, but for safety and
> > performance reasons (I've built by own RODict and ROList modeled after
> > dictproxy):
> >
> > - Global, but immutable containers, e.g. as class members
>
> I attached type_final.patch to the issue #14162 to demonstrate how
> frozendict can be used to implement a "read-only" type. Last version:
> http://bugs.python.org/file24696/type_final.patch

Oh, hmm. I rather meant something like that:

"""
class Foo:
some_mapping = frozendict(
blah=1, blub=2
)

or as a variant:

def zonk(some_default=frozendict(...)):
...

or simply a global object:

baz = frozendict(some_immutable_mapping)
"""

I'm not sure about your final types. I'm using __slots__ = () for such things 
(?)

nd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Nick Coghlan]

On Thu, Mar 1, 2012 at 7:29 PM, André Malo  wrote:
> - Caching. My data container objects (say, resultsets from a db or something)
>  usually inherit from list or dict (sometimes also set) and are cached
>  heavily. In order to ensure that they are not modified (accidentially), I
>  have to choices: deepcopy or immutability. deepcopy is so expensive, that
>  it's often cheaper to just leave out the cache. So I use immutability. (oh
>  well, the objects are further restricted with __slots__)

Speaking of caching - functools.lru_cache currently has to do a fair
bit of work in order to correctly cache keyword arguments. It's
obviously a *solvable* problem even without frozendict in the
collections module (it just stores the dict contents as a sorted tuple
of 2-tuples), but it would still be interesting to compare the
readability, speed and memory consumption differences of a version of
lru_cache that used frozendict to cache the keyword arguments instead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Serhiy Storchaka]

01.03.12 11:11, Victor Stinner написав(ла):

You can redefine dict.__setitem__.

Ah? It doesn't work here.


dict.__setitem__=lambda key, value: None

Traceback (most recent call last):
   File "", line 1, in
TypeError: can't set attributes of built-in/extension type 'dict'


Hmm, yes, it's true. It was too presumptuous of me to believe that you 
have not considered such simple approach.


But I will try to suggest another approach. `frozendict` inherits from 
`dict`, but data is not stored in the parent, but in the internal 
dictionary. And even if dict.__setitem__ is used, it will have no 
visible effect.


  class frozendict(dict):
  def __init__(self, values={}):
  self._values = dict(values)
  def __getitem__(self, key):
  return self._values[key]
  def __setitem__(self, key, value):
  raise TypeError ("expect dict, got frozendict")
  ...


 >>> a = frozendict({1: 2, 3: 4})
 >>> a[1]
 2
 >>> a[5]
   Traceback (most recent call last):
   File "", line 1, in 
   File "", line 5, in __getitem__
 KeyError: 5
 >>> a[5] = 6
 Traceback (most recent call last):
   File "", line 1, in 
   File "", line 7, in __setitem__
 TypeError: expect dict, got frozendict
 >>> dict.__setitem__(a, 5, 6)
 >>> a[5]
 Traceback (most recent call last):
   File "", line 1, in 
   File "", line 5, in __getitem__
 KeyError: 5

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 416: Add a frozendict builtin type

[Paul Moore]

On 1 March 2012 12:08, Victor Stinner  wrote:
> New try:
>
> "A frozendict is a read-only mapping: a key cannot be added nor
> removed, and a key is always mapped to the same value. However,
> frozendict values can be mutable (not hashable). A frozendict is
> hashable and so immutable if and only if all values are hashable
> (immutable)."

I'd suggest you don't link immutability and non-hashability so
tightly. Misbehaved objects can be mutable but hashable:

>>> class A:
...   def __init__(self,a):
... self.a = a
...   def __hash__(self):
... return 12
...
>>> a = A(1)
>>> hash(a)
12
>>> a.a=19
>>> hash(a)
12

Just avoid using the term "immutable" at all:

"A frozendict is a read-only mapping: a key cannot be added nor
removed, and a key is always mapped to the same value. However,
frozendict values can be mutable. A frozendict is hashable if and
only if all values are hashable."

I realise this is a weaker statement than you'd like to give
(immutability seems to be what people *really* think they want when
they talk about frozen objects), but don't promise immutability if
that's not what you're providing.

More specifically, I'd hate to think that someone for whom security is
an issue would see your original description and think they could use
a frozendict and get safety, only to find their system breached
because of a class like A above. The same could happen to people who
want to handle thread safety via immutable objects, who could also end
up with errors if misbehaving classes found their way into an
application.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Paul Moore]

On 1 March 2012 12:37, Yury Selivanov  wrote:
> Actually I find fronzendict concept quite useful.  We also have an
> implementation in our framework, and we use it, for instance, in
> http request object, for parsed arguments and parsed forms, which
> values shouldn't be ever modified once parsed.

The question isn't so much whether it's useful, as whether it's of
sufficiently general use to warrant putting it into the core language
(not even the stdlib, but the C core!). The fact that you have an
implementation of your own, actually indicates that not having it in
the core didn't cause you any real problems.

Remember - the bar for core acceptance is higher than just "it is
useful". I'm not even sure I see a strong enough case for frozendict
being in the standard library yet, let alone in the core.

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Serhiy Storchaka]

01.03.12 11:29, André Malo написав(ла):

- Caching. My data container objects (say, resultsets from a db or something)
   usually inherit from list or dict (sometimes also set) and are cached
   heavily. In order to ensure that they are not modified (accidentially), I
   have to choices: deepcopy or immutability. deepcopy is so expensive, that
   it's often cheaper to just leave out the cache. So I use immutability. (oh
   well, the objects are further restricted with __slots__)


This is the first rational use of frozendict that I see. However, a deep 
copy is still necessary to create the frozendict. For this case, I 
believe, would be better to "freeze" dict inplace and then copy-on-write it.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

On Thursday 01 March 2012 15:17:35 Serhiy Storchaka wrote:
> 01.03.12 11:29, André Malo написав(ла):
> > - Caching. My data container objects (say, resultsets from a db or
> > something) usually inherit from list or dict (sometimes also set) and are
> > cached heavily. In order to ensure that they are not modified
> > (accidentially), I have to choices: deepcopy or immutability. deepcopy is
> > so expensive, that it's often cheaper to just leave out the cache. So I
> > use immutability. (oh well, the objects are further restricted with
> > __slots__)
>
> This is the first rational use of frozendict that I see. However, a deep
> copy is still necessary to create the frozendict. For this case, I
> believe, would be better to "freeze" dict inplace and then copy-on-write
> it.

In my case it's actually a half one. The data mostly comes from memcache ;) 
I'm populating the object and then I'm done with it. People wanting to modify 
it, need to copy it, yes. OTOH usually a shallow copy is enough (here).

Funnily my ROList actually provides a "sorted" method instead of "sort" in 
order to create a sorted copy of the list.

nd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

> But I will try to suggest another approach. `frozendict` inherits from
> `dict`, but data is not stored in the parent, but in the internal
> dictionary. And even if dict.__setitem__ is used, it will have no visible
> effect.
>
>  class frozendict(dict):
>      def __init__(self, values={}):
>          self._values = dict(values)
>      def __getitem__(self, key):
>          return self._values[key]
>      def __setitem__(self, key, value):
>          raise TypeError ("expect dict, got frozendict")
>      ...

I would like to implement frozendict in C to be able to pass it to
PyDict_GetItem(), PyDict_SetItem() and PyDict_DelItem(). Using such
Python implementation, you would get surprising result:

d = frozendict()
dict.__setitem__(d, 'x', 1) # this is what Python does internally when
it expects a dict (e.g. in ceval.c for __builtins__)
'x' in d => False

(Python is not supposed to use the PyDict API if the object is a dict
subclass, but PyObject_Get/SetItem.)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

>> > Here are my real-world use cases. Not for security, but for safety and
>> > performance reasons (I've built by own RODict and ROList modeled after
>> > dictproxy):
>> >
>> > - Global, but immutable containers, e.g. as class members
>>
>> I attached type_final.patch to the issue #14162 to demonstrate how
>> frozendict can be used to implement a "read-only" type. Last version:
>> http://bugs.python.org/file24696/type_final.patch
>
> Oh, hmm. I rather meant something like that:
>
> """
> class Foo:
>    some_mapping = frozendict(
>        blah=1, blub=2
>    )
> or as a variant:
>
> def zonk(some_default=frozendict(...)):
>    ...
> or simply a global object:
>
> baz = frozendict(some_immutable_mapping)
> """

Ah yes, frozendict is useful for such cases.

> I'm not sure about your final types. I'm using __slots__ = () for such things

You can still replace an attribute value if a class defines __slots__:

>>> class A:
...   __slots__=('x',)
...   x = 1
...
>>> A.x=2
>>> A.x
2

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

On Thursday 01 March 2012 15:54:01 Victor Stinner wrote:

> > I'm not sure about your final types. I'm using __slots__ = () for such
> > things
>
> You can still replace an attribute value if a class defines __slots__:
> >>> class A:
>
> ...   __slots__=('x',)
> ...   x = 1
> ...
>
> >>> A.x=2
> >>> A.x
>
> 2

Ah, ok, I missed that. It should be fixable with a metaclass. Not very nicely, 
though.

nd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] EuroPython 2012: Call for Proposal is Open! [Please spread the word]

[Francesco Pallanti]

Hi guys,
I'm Francesco and I am writing on behalf of EuroPython Staff
(www.europython.eu). We are happy to announce that the Call for
Proposals is now officially open! 

DEADLINE FOR PROPOSALS: MARCH 18TH, 23:59:59 CET

For those who have never been at EuroPython (or similar conferences)
before, the Call for Proposals is the period in which the organizers ask
the community to submit proposals for talks to be held at the
conference.

Further details about Call for Proposal are online here:
http://ep2012.europython.eu/call-for-proposals/

EuroPython is a conference run by the community for the community: the
vast majority of talks that are presented at the conference will be
proposed, prepared and given by members of the Python community itself.

And not only that: the process that selects the best talks among all the
proposals will also be public and fully driven by the community: it's
called Community Voting, and will begin right after the Call for
Proposals ends.

CFP: Talks, Hands-On Trainings and Posters
--

We're looking for proposals on every aspect of Python: programming from
novice to advanced levels, applications and frameworks, or how you have
been involved in introducing Python into your organisation.

There are three different kind of contribution that you can present at
EuroPython:
- Regular talk. These are standard "talk with slides", allocated in
slots of 45, 60 or 90 minutes, depending on your preference and
scheduling constraints. A Q&A session is held at the end of the talk.
- Hands-on training. These are advanced training sessions for a smaller
audience (10-20 people), to dive into the subject with all details.
These sessions are 4-hours long, and the audience will be strongly
encouraged to bring a laptop to experiment. They should be prepared with
less slides and more source code.
- Posters. Posters are a graphical way to describe a project or a
technology, printed in large format; posters are exhibited at the
conference, can be read at any time by participants, and can be
discussed face to face with their authors during the poster session. We
will take care of printing the posters too, so don't worry about
logistics.

More details about Call for Proposal are online here:
http://ep2012.europython.eu/call-for-proposals/

Don't wait for the last day
---

If possible, please avoid submitting your proposals on the last day. It
might sound a strange request, but last year about 80% of the proposals
were submitted in the last 72 hours. This creates a few problems for
organizers because we can't have a good picture of the size of the
conference until that day.

Remember that proposals are fully editable at any time, even after the
Call for Proposals ends. You just need to login on the website, go to
the proposal page (linked from your profile page), and click the Edit
button.

First-time speakers are especially welcome; EuroPython is a community
conference and we are eager to hear about your experience. If you have
friends or colleagues who have something valuable to contribute, twist
their arms to tell us about it!

We are a conference run by the community for the community. Please help
to spread the word by distributing this announcement to colleagues,
mailing lists, your blog, Web site, and through your social networking
connections. 

All the best,


-- 
Francesco Pallanti - fpalla...@develer.com
Develer S.r.l. - http://www.develer.com/
.software  .hardware  .innovation
Tel.: +39 055 3984627 - ext.: 215

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Stefan Krah]

Merlijn van Deen  wrote:
> Another cause for this is the painful merging in most version control systems.
> I'm guessing you all know the pain of 'svn merge' - and there are a lot of
> projects still using SVN or even CVS.
>
> As such, you need to impose the discipline to always apply changes to both
> branches. This is a reasonable thing for larger projects, but it is generally
> harder to implement it for smaller projects, as you're already lucky people 
> are
> actually contributing.

What you say is all true, but I wonder if the additional work is really
that much of a problem.

Several people have said here that applying changes to both versions
becomes second nature, and this is also my experience. While mercurial
may be nicer, svnmerge.py isn't that bad.


Projects have different needs and priorities. From my own experience with
cdecimal I can positively say that the amount of work required to keep
two branches [1] in sync is completely dwarfed by first figuring out
what to write and then implementing it correctly.

After doing all that, the actual synchronization work feels like a vacation.


Another aspect, which may be again cdecimal-specific, is that keeping
2.5 compatibility is *at least* as bothersome as supporting 2.6/2.7 and 3.x.


As an example for a pretty large project, it looks like Antoine is making
good progress with Twisted:

https://bitbucket.org/pitrou/t3k/wiki/Home


I certainly can't say what's possible or best for other projects. I do
think though that choosing the separate branches strategy will pay off
eventually (at the very latest when Python-2.7 will reach the status
that Python-1.5 currently has).


Stefan Krah


[1] I don't even use two branches but 2.c/3.c and 2.py/3.py file name patterns.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Antoine Pitrou]

On Thu, 1 Mar 2012 16:31:14 +0100
Stefan Krah  wrote:
> 
> As an example for a pretty large project, it looks like Antoine is making
> good progress with Twisted:
> 
> https://bitbucket.org/pitrou/t3k/wiki/Home

Well, to be honest, "making good progress" currently means "bored and
not progressing at all" :-) But that's not due to the strategy I
adopted, only to the sheer amount of small changes needed, and lack of
immediate motivation to continue this work.

However, merging actually ended up easier than I expected. The last
time, merging one month's worth of upstream changes took me around one
hour (including fixing additional tests and regressions).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Stefan Krah]

Lennart Regebro  wrote:
> I also don't agree with the claim that a py3 version using 2to3 is a
> "second class citizen". You need to adopt the Python 2 code to Python
> 3 in that case too, and none of the overrules the other.

That's a fair point. Then of course *both* versions do not use their full
potential, but that is strongly related to the "using all (new) features"
item in the list.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Barry Warsaw]

On Mar 01, 2012, at 04:42 PM, Antoine Pitrou wrote:

>Well, to be honest, "making good progress" currently means "bored and
>not progressing at all" :-) But that's not due to the strategy I
>adopted, only to the sheer amount of small changes needed, and lack of
>immediate motivation to continue this work.

For any porting strategy, the best thing to do is to get as many changes into
upstream as possible that prepares the way for Python 3 support.  For example,
when I did the dbus-python port, upstream (rightly so) rejected my big
all-together-now patch.

Instead, we took a number of smaller steps, many of which were incorporated
before the Python 3 support landed.  These included:

- Agreeing to Python 2.6 as a minimum base
- #include  and global PyString_* -> PyBytes_* conversion
- (yes) adding future imports for unicode_literals, unadorning unicodes and
  adding b'' prefixes where necessary
- fixing except clauses to use 'as'
- removing L suffix on integer literals
- lots of other little syntactic nits

You could add to that things like print functions (although IIRC dbus-python
had few if any of these), etc.  So really, it was the same strategy as any
porting process, but the key was breaking these up into reviewable chunks that
could be applied while still keeping the code base Python 2 only.

I really do think that to the extent that you can do that kind of thing, you
may end up with essentially Python 3 support without even realizing it. :)

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[Antoine Pitrou]

On Thu, 1 Mar 2012 11:24:19 -0500
Barry Warsaw  wrote:
> 
> I really do think that to the extent that you can do that kind of thing, you
> may end up with essentially Python 3 support without even realizing it. :)

That's unlikely. Twisted processes bytes data a lot, and the bytes
indexing behaviour of 3.x is a chore for porting.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Guido van Rossum]

On Thu, Mar 1, 2012 at 2:01 AM, Victor Stinner
 wrote:
>>> The main idea of pysandbox is to reuse most of CPython but hide
>>> "dangerous" functions and run untrusted code in a separated namespace.
>>> The problem is to create the sandbox and ensure that it is not
>>> possible to escape from this sandbox. pysandbox is still a
>>> proof-of-concept, even if it works pretty well for short dummy
>>> scripts. But pysandbox is not ready for real world programs.
>>
>> I hope you have studied (recent) history. Sandboxes in Python
>> traditionally have not been secure. Read the archives for details.
>
> The design of pysandbox makes it difficult to implement. It is mostly
> based on blacklist, so any omission would lead to a vulnerability. I
> read the recent history of sandboxes and see other security modules
> for Python, and I don't understand your reference to  "Sandboxes in
> Python traditionally have not been secure." There is no known
> vulnerability in pysandbox, did I miss something? (there is only a
> limitation on the dict API because of the lack of frozendict.)
>
> Are you talking about rexec/Bastion? (which cannot be qualified as "recent" 
> :-))
>
> pysandbox limitations are documented in its README file:
>
> << pysandbox is a sandbox for the Python namespace, not a sandbox between 
> Python
> and the operating system. It doesn't protect your system against Python
> security vulnerabilities: vulnerabilities in modules/functions available in
> your sandbox (depend on your sandbox configuration). By default, only few
> functions are exposed to the sandbox namespace which limits the attack 
> surface.
>
> pysandbox is unable to limit the memory of the sandbox process: you have to 
> use
> your own protection. >>
>
> Hum, I am also not sure that pysandbox "works" with threads :-) I mean
> that enabling pysandbox impacts all running threads, not only one
> thread, which can cause issues. It should also be mentioned.
>
> PyPy sandbox has a different design: it uses a process with no
> priviledge, all syscalls are redirected to another process which apply
> security checks to each syscall.
> http://doc.pypy.org/en/latest/sandbox.html
>
> See also the seccomp-nurse project, a generic sandbox using Linux SECCOMP:
> http://chdir.org/~nico/seccomp-nurse/
>
> See also pysandbox README for a list of other Python security modules.

Hm. I can't tell what the purpose of a sandbox is from what you quote
from your own README here (and my cellphone tethering is slow enough
that clicking on the links doesn't work right now).

The sandboxes I'm familiar with (e.g. Google App Engine) are intended
to allow untrusted third parties to execute (more or less) arbitrary
code while strictly controlling which resources they can access. In
App Engine's case, an attacker who broke out of the sandbox would have
access to the inside of Google's datacenter, which would obviously be
bad -- that's why Google has developed its own sandboxing
technologies.

I do know that I don't feel comfortable having a sandbox in the Python
standard library or even recommending a 3rd party sandboxing solution
-- if someone uses the sandbox to protect a critical resource, and a
hacker breaks out of the sandbox, the author of the sandbox may be
held responsible for more than they bargained for when they made it
open source. (Doesn't an open source license limit your
responsibility? Who knows. AFAIK this question has not gotten to court
yet. I wouldn't want to have to go to court over it.)

I wasn't just referring of rexec/Bastion (though that definitely
shaped my thinking about this issue; much more recently someone (Tal,
I think was his name?) tried to come up with a sandbox and every time
he believed he had a perfect solution, somebody found a loophole.
(Hm..., you may have been involved that time yourself. :-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Spreading the Python 3 religion

[R. David Murray]

On Thu, 01 Mar 2012 17:24:31 +0100, Antoine Pitrou  wrote:
> On Thu, 1 Mar 2012 11:24:19 -0500
> Barry Warsaw  wrote:
> > 
> > I really do think that to the extent that you can do that kind of thing, you
> > may end up with essentially Python 3 support without even realizing it. :)
> 
> That's unlikely. Twisted processes bytes data a lot, and the bytes
> indexing behaviour of 3.x is a chore for porting.

The dodges you have to use work fine in python2 as well, though, so
I think Barry's point stands, even if it does make the python2 code a
bit uglier...but not as bad as the 2.5 exception hacks.  Still, I'll
grant that it would be a harder sell to upstream than the changes Barry
mentioned.  On the other hand, it's not like the code will get
*prettier* once you drop Python2 support :(.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Guido van Rossum]

I noticed there were some complaints about unnecessarily offensive
language in PEP 414. Have those passages been edited to everyone's
satisfaction?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

> In App Engine's case, an attacker who broke out of the sandbox would have
> access to the inside of Google's datacenter, which would obviously be
> bad -- that's why Google has developed its own sandboxing
> technologies.

This is not specific to Google: if an attacker breaks a sandbox,
he/she has access to everything. Depending on how the sandbox is
implemented, you have more or less code to audit.

pysandbox disables introspection in Python and create an empty
namespace to reduce as much as possible the attack surface. You are to
be very careful when you add a new feature/function and it is complex.

> I do know that I don't feel comfortable having a sandbox in the Python
> standard library or even recommending a 3rd party sandboxing solution

frozendict would help pysandbox but also any security Python module,
not security, but also (many) other use cases ;-)

> I wasn't just referring of rexec/Bastion (though that definitely
> shaped my thinking about this issue; much more recently someone (Tal,
> I think was his name?) tried to come up with a sandbox and every time
> he believed he had a perfect solution, somebody found a loophole.
> (Hm..., you may have been involved that time yourself. :-)

pysandbox is based on tav's approach, but it is more complete and
implement more protections. It is also more functional (you have more
available functions and features).

I challenge anyone to try to break pysandbox!

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Serhiy Storchaka]

01.03.12 16:47, André Malo написав(ла):
> On Thursday 01 March 2012 15:17:35 Serhiy Storchaka wrote:
>> This is the first rational use of frozendict that I see. However, a deep
>> copy is still necessary to create the frozendict. For this case, I
>> believe, would be better to "freeze" dict inplace and then copy-on-write
>> it.
> In my case it's actually a half one. The data mostly comes from memcache ;)
> I'm populating the object and then I'm done with it. People wanting to modify
> it, need to copy it, yes. OTOH usually a shallow copy is enough (here).

What if people modify dicts in deep?

 a = frozendict({1: {2: 3}})
 b = a.copy()
 c = a.copy()
 assert b[1][2] == 3
 c[1][2] = 4
 assert b[1][2] == 4

You need to copy incoming dict in depth.

 def frozencopy(value):
 if isinstance(value, list):
 return tuple(frozencopy(x) for x in value)
 if isinstance(value, dict):
 return frozendict((frozencopy(k), frozencopy(v)) for k, v in 
value.items())
 return value # I'm lucky

And when client wants to modify the result in depth it should call 
"unfrozencopy". Using frozendict profitable only when multiple clients are 
reading the result, but not modify it. Copy-on-write would help in all cases 
and would simplify the code. But this is a topic for python-ideas, sorry.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Paul Moore]

On 1 March 2012 17:44, Victor Stinner  wrote:

> I challenge anyone to try to break pysandbox!

Can you explain precisely how a frozendict will help pysandbox? Then
I'll be able to beat this challenge :-)

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Guido van Rossum]

On Thu, Mar 1, 2012 at 9:44 AM, Victor Stinner  wrote:
> frozendict would help pysandbox but also any security Python module,
> not security, but also (many) other use cases ;-)

Well, let's focus on the other use cases, because to me the sandbox
use case is too controversial (never mind how confident you are :-).

I like thinking through the cache use case a bit more, since this is a
common pattern. But I think it would be sufficient there to prevent
accidental modification, so it should be sufficient to have a dict
subclass that overrides the various mutating methods: __setitem__,
__delitem__, pop(), popitem(), clear(), setdefault(), update().
Technically also __init__() -- although calling __init__() on an
existing object can hardly be called an accident. As was pointed out
this is easy to circumvent, but (together with a reminder in the
documentation) should be sufficient to avoid mistakes. I imagine
someone who actively wants to mess with the cache can probably also
reach into the cache implementation directly.

Also don't underestimate the speed of a shallow dict copy.

What other use cases are there? (I have to agree with the folks
pushing back hard. Even demonstrated repeated requests for a certain
feature do not prove a need -- it's quite common for people who are
trying to deal with some problem to go down the wrong rabbit hole in
their quest for a solution, and ending up thinking they need a certain
feature while completely overlooking a much simpler solution.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Sandboxing Python

[Victor Stinner]

Hi,

The frozendict discussion switched somewhere to sandboxing, and so I
prefer to start a new thread.

There are various ways to implement a sandbox, but I would like to
expose here how I implemented pysandbox to have your opinion.
pysandbox is written to execute quickly a short untrusted function in
a sandbox and then continue the normal execution of the program. It is
possible to "enable" the sandbox, but also later to "disable" it. It
is written for Python using only one thread and one process.

To create a sandbox, pysandbox uses various protections. The main idea
is to create an empty namespace and ensure that it is not possible to
use objects added into the namespace for escaping from the sandbox.
pysandbox only uses one thread and one process and so it doesn't
replace the existing trusted namespace, but create a new one. The
security of pysandbox depends on the sandbox namespace sealing.

I don't want to integrate pysandbox in CPython because I am not yet
conviced that the approach is secure by design. I am trying to patch
Python to help the implementation of Python security modules and of
read-only proxies.

You can find below the list of protections implemented in pysandbox.
Some of them are implemented in C.

I challenge anymore to break pysandbox! I would be happy if anyone
breaks it because it would make it more stronger.

https://github.com/haypo/pysandbox/
http://pypi.python.org/pypi/pysandbox

Namespace
=

 * Make builtins read only
 * Remove function attribute:

   * frame.f_locals
   * function.func_closure/__closure__
   * function.func_defaults/__defaults__
   * function.func_globals/__globals__
   * type.__subclasses__
   * builtin_function.__self__

 * Workaround the lack of frozendict, remove dict attributes:

   *__init__
   * clear
   * __delitem__
   * pop
   * popitem
   * setdefault
   * __setitem__
   * update

 * Create a proxy for objects injected to the sandbox namespace and for
   the result of functions (the result of callable objects is also proxified)


Generic
===

Remove all builtin symbols not in the whitelist.


Features


import
--

 * Replace __import__ function to use an import whitelist

Filesystem
--

 * Replace open and file functions to deny access to the filesystem

Exit


 * Replace exit function
 * Remove SystemExit builtin exception

Standard input/output
-

 * Replace sys.stdin, sys.stdout and sys.stderr


Bytecode


Execute arbitrary bytecode may crash Python, or lead to execution of arbitrary
(machine) code.

 * Patch code.__new__()
 * Remove attributes:

   * function.func_code/__code__
   * frame.f_code
   * generator.gi_code

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

>> I challenge anyone to try to break pysandbox!
>
> Can you explain precisely how a frozendict will help pysandbox? Then
> I'll be able to beat this challenge :-)

See this email:
http://mail.python.org/pipermail/python-dev/2012-February/117011.html

The issue #14162 has also two patches: one to make it possible to use
frozendict for __builtins__, and another one to create read-only types
(which is more a proof-of-concept).
http://bugs.python.org/issue14162

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Barry Warsaw]

On Mar 01, 2012, at 09:42 AM, Guido van Rossum wrote:

>I noticed there were some complaints about unnecessarily offensive
>language in PEP 414. Have those passages been edited to everyone's
>satisfaction?

Not yet, but I believe Nick volunteered to do a rewrite.

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Compiling Python on Linux with Intel's icc

[Alex Leach]

Dear Python Devs,

I've been attempting to compile a fully functional version of Python 2.7 using 
Intel's C compiler, having built supposedly optimal versions of numpy and 
scipy, using Intel Composer XE and Intel's Math Kernel Library. I can build a 
working Python binary, but I'd really appreciate if someone could check my 
compile options, and perhaps suggest ways I could further optimise the build.

*** COMPILE FAILURE - ffi64.c ***

I've managed to compile everything in the python distribution except for 
Modules/_ctypes/libffi/src/x86/ffi64.c. So to get the compilation to actually 
work, I've had to use the config option '--with-system-ffi'. If someone could 
suggest a patch for ffi64.c, I'd happily test it, as I've been unable to fix 
the 
code myself! The problem is with register_args, which uses GCC's __int128_t, 
but this doesn't exist when using icc.

The include guard to use could be:-
#ifdef __INTEL_COMPILER
...
#else
...
#endif

I've tried using this guard around the register_args struct, at the top of 
ffi64.c, and where I see register_args used, around lines 592-616, according to 
the suggestion at http://software.intel.com/en-
us/forums/showthread.php?t=56652, but have been unable to get a working 
solution... A patch would be appreciated!

*** Tests ***

After compilation, there's a few tests that are consistently failing, mainly 
involved with floating point precision: test_cmath, test_math and test_float.  
Also, I wrote a very short script to test the time of for loop execution and 
integer multiplication. This script (below) has nearly always completed faster 
using the default Ubuntu Python rather than my own build.

Obviously, I was hoping to get a faster python, but the size of the final 
binary is almost twice the size of the default Ubuntu version (5.2MB cf. 
2.7MB), which I thought might cause a startup overhead that leads to slower 
execution times when running such a basic script.


*** TEST SCRIPT ***
$ cat ~/bin/timetest.py

RANGE = 1

print "running {0}^2 = {1} for loop iterations".format( RANGE,RANGE**2 )

for i in xrange(RANGE):
for j in xrange(RANGE):
i * j


*** TIMES ***

## ICC-compiled python ##
$ time ./python ~/bin/timetest.py
running 1^2 = 1 for loop iterations

real0m2.767s
user0m2.720s
sys 0m0.008s


## System python ##
$ time python ~/bin/timetest.py
running 1^2 = 1 for loop iterations

real0m2.781s
user0m2.776s
sys 0m0.000s

Oh... My python appears to run faster than gcc's now - checked this a few 
times now, mine's staying faster... :) I've compiled and re-compiled python 
dozens of times now, but it's still failing some tests...


*** Build Environment ***

Ubuntu 10.10 server kernel (`uname -r`=3.0.0-16-server) with KDE 4.7.4

$ tail ~/.bashrc

 Custom Commands
export PATH=$PATH:/usr/local/cuda/bin:$HOME/bin
export PYTHONPATH=$HOME/bin:/usr/lib/pymodules/python2.7
export PYTHONSTARTUP=$HOME/.pystartup
export 
LD_LIBRARY_PATH=/lib64:/usr/lib64:/usr/local/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib
# Load Intel compiler and library variables.
source /usr/intel/bin/compilervars.sh intel64
source /usr/intel/impi/4.0.3/bin/mpivars.sh intel64
source /usr/intel/tbb/bin/tbbvars.sh intel64

$ env | grep 'PATH\|FLAGS'
MANPATH=/usr/intel/impi/4.0.3.008/man:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/intel/impi/4.0.3.008/man:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/intel/impi/4.0.3.008/man:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/intel/composer_xe_2011_sp1.9.293/man/en_US:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/intel/man:::
LIBRARY_PATH=/usr/intel/composer_xe_2011_sp1.9.293/tbb/lib/intel64//cc4.1.0_libc2.4_kernel2.6.16.21:/usr/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/ipp/../compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/ipp/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/mkl/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/tbb/lib/intel64//cc4.1.0_libc2.4_kernel2.6.16.21
FPATH=/usr/intel/composer_xe_2011_sp1.9.293/mkl/include:/usr/intel/composer_xe_2011_sp1.9.293/mkl/include
LD_LIBRARY_PATH=/usr/intel/composer_xe_2011_sp1.9.293/tbb/lib/intel64//cc4.1.0_libc2.4_kernel2.6.16.21:/usr/intel/impi/4.0.3.008/ia32/lib:/usr/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/ipp/../compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/ipp/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/compiler/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/mkl/lib/intel64:/usr/intel/composer_xe_2011_sp1.9.293/tbb/lib/intel64//cc4.1.0_libc2.4_kernel2.6.16.21:/biol/arb/lib:/lib64:/usr/lib64:/usr/local/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/intel/composer_xe_2011_sp1.9.293/debugger/lib/intel64:/usr/in

Re: [Python-Dev] PEP 414

[Vinay Sajip]

Guido van Rossum  python.org> writes:

> I noticed there were some complaints about unnecessarily offensive
> language in PEP 414. Have those passages been edited to everyone's
> satisfaction?

I'm not sure if Nick has finished his updates, but I for one would like to see
some improvements in a few places:

"Many thought that the unicode_literals future import might make a common source
possible, but it turns out that it's doing more harm than good."

Rather than talking about it doing more harm than good, it would be better to
say that unicode_literals is not the best solution in some scenarios
(specifically, WSGI, but any other scenarios can also be mentioned). The "more
harm than good" is not true in all scenarios, but as it's worded now, it seems
like it is always a bad approach.

"(either by having a u function that marks things as unicode without future
imports or the inverse by having a n function that marks strings as native).
Unfortunately, this has the side effect of slowing down the runtime performance
of Python and makes for less beautiful code."

The use of u() and n() are not equivalent in the sense that n() only has to be
used when unicode_literals are in effect, and the incidence of n() calls in an
application would be much lower than using u() in the absence of
unicode_literals. In at least some cases, it is possible that some of the APIs
which fail unless native strings are provided may be broken (e.g. some database
adapters expect datetimes in ISO format as native strings, where there is no
apparent reason why they couldn't accept them as text).

As far as "less beautiful" code is concerned, it's subjective: I see nothing
especially ugly about 'xxx' for text, and certainly don't find u'xxx' "more"
beautiful - and I doubt if I'm the only person with that view. The point about
the added cognitive burden of semantic-changing __future__ imports is, however,
quite valid.

"As it stands, when chosing between 2.7 and Python 3.2, Python 3 is currently
not the best choice for certain long-term investments, since the ecosystem is
not yet properly developed, and libraries are still fighting with their API
decisions for Python 3."

This looks to become a self-fulfilling prophecy, if you take it seriously. You
would expect that, if Python 3 is the future of Python, then Python 3 is
*precisely* the choice for *long*-term investments. The ecosystem is not yet
fully developed, true: but that is because some people aren't ready to grasp the
nettle and undergo the short-term pain required to get things in order. By
"things", I mean places in existing 2.x code where no distinction was made
between bytes and text, which you could get away with because of 2.x's forgiving
nature. Whether you're using unicode_literals and 'xxx' or u'xxx', these things
will need to be sorted out, and the syntax element is only one possible focus.

If that entire sentence is removed, it does the PEP no harm, and the PEP will
antagonise fewer people.

"A valid point is that this would encourage people to become dependent on Python
3.3 for their ports. Fortunately that is not a big problem since that could be
fixed at installation time similar to how many projects are currently invoking
2to3 as part of their installation process."

Yes, but avoiding the very pain of running 2to3 is what (at least in part)
motivates the PEP in the first place. This appears to be moving the pain that
2.x developers feel when trying to move to 3.x, to people who want to support
3.2 and 3.3 and 2.6+ in the same codebase.

"For Python 3.1 and Python 3.2 (even 3.0 if necessary) a simple on-installation
hook could be provided that tokenizes all source files and strips away the
otherwise unnecessary u prefix at installation time."

There's some confusion about this hook - The PEP calls it an on-installation
hook (like 2to3) but Nick said it was an import-time hook. I'm more comfortable
with the latter - it has a chance of providing an acceptable performance for a
large codebase, as it will only kick in when .py files are newer than their
.pyc. A 2to3 like hook, when working with a large codebase like Django, is
likely to be about as painful as people are finding 2to3 now (when used in an
edit-test-edit-test workflow).

"Possible Downsides" does not mention any possible adverse impact on single
codebase for 3.2/3.3, which I mention only because it's still not clear how the
hook which is to make 3.2 development easier will work (in terms of its impact
on development workflow).

In the section on "Modernizing code",

"but to make strings cheap for both 2.x and 3.x it is nearly impossible. The way
it currently works is by abusing the unicode-escape codec on Python 2.x native
strings."

IIUC, the unicode-escape codec is only needed if you don't use unicode_literals
- am I wrong about that? How are strings not equally cheap (near enough) on 2.x
and 3.x if you use unicode_literals?

In the "Runtime overhead of wrappers", the times may be valid, but a rider
shoul

Re: [Python-Dev] Compiling Python on Linux with Intel's icc

[Stefan Krah]

Hi,

Alex Leach  wrote:
> I've managed to compile everything in the python distribution except for
> Modules/_ctypes/libffi/src/x86/ffi64.c.

There is an issue for this:

http://bugs.python.org/issue4130


> After compilation, there's a few tests that are consistently failing, mainly
> involved with floating point precision: test_cmath, test_math and test_float.

I think you have to compile with "-fp-model strict".


In general, please submit suspected bugs on http://bugs.python.org/
(after searching the archives) and post things like speed comparisons on
python-l...@python.org.


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Yury Selivanov]

Vinay,

Thank you for the comprehensive summary.  Big +1.  I really
do hope that Nick and Armin will rectify the PEP.  Otherwise,
many of its points are moot, and we need to raise a question
of rejecting it somehow.

On 2012-03-01, at 2:00 PM, Vinay Sajip wrote:

> Guido van Rossum  python.org> writes:
> 
>> I noticed there were some complaints about unnecessarily offensive
>> language in PEP 414. Have those passages been edited to everyone's
>> satisfaction?
> 
> I'm not sure if Nick has finished his updates, but I for one would like to see
> some improvements in a few places:
> 
> "Many thought that the unicode_literals future import might make a common 
> source
> possible, but it turns out that it's doing more harm than good."
> 
> Rather than talking about it doing more harm than good, it would be better to
> say that unicode_literals is not the best solution in some scenarios
> (specifically, WSGI, but any other scenarios can also be mentioned). The "more
> harm than good" is not true in all scenarios, but as it's worded now, it seems
> like it is always a bad approach.
> 
> "(either by having a u function that marks things as unicode without future
> imports or the inverse by having a n function that marks strings as native).
> Unfortunately, this has the side effect of slowing down the runtime 
> performance
> of Python and makes for less beautiful code."
> 
> The use of u() and n() are not equivalent in the sense that n() only has to be
> used when unicode_literals are in effect, and the incidence of n() calls in an
> application would be much lower than using u() in the absence of
> unicode_literals. In at least some cases, it is possible that some of the APIs
> which fail unless native strings are provided may be broken (e.g. some 
> database
> adapters expect datetimes in ISO format as native strings, where there is no
> apparent reason why they couldn't accept them as text).
> 
> As far as "less beautiful" code is concerned, it's subjective: I see nothing
> especially ugly about 'xxx' for text, and certainly don't find u'xxx' "more"
> beautiful - and I doubt if I'm the only person with that view. The point about
> the added cognitive burden of semantic-changing __future__ imports is, 
> however,
> quite valid.
> 
> "As it stands, when chosing between 2.7 and Python 3.2, Python 3 is currently
> not the best choice for certain long-term investments, since the ecosystem is
> not yet properly developed, and libraries are still fighting with their API
> decisions for Python 3."
> 
> This looks to become a self-fulfilling prophecy, if you take it seriously. You
> would expect that, if Python 3 is the future of Python, then Python 3 is
> *precisely* the choice for *long*-term investments. The ecosystem is not yet
> fully developed, true: but that is because some people aren't ready to grasp 
> the
> nettle and undergo the short-term pain required to get things in order. By
> "things", I mean places in existing 2.x code where no distinction was made
> between bytes and text, which you could get away with because of 2.x's 
> forgiving
> nature. Whether you're using unicode_literals and 'xxx' or u'xxx', these 
> things
> will need to be sorted out, and the syntax element is only one possible focus.
> 
> If that entire sentence is removed, it does the PEP no harm, and the PEP will
> antagonise fewer people.
> 
> "A valid point is that this would encourage people to become dependent on 
> Python
> 3.3 for their ports. Fortunately that is not a big problem since that could be
> fixed at installation time similar to how many projects are currently invoking
> 2to3 as part of their installation process."
> 
> Yes, but avoiding the very pain of running 2to3 is what (at least in part)
> motivates the PEP in the first place. This appears to be moving the pain that
> 2.x developers feel when trying to move to 3.x, to people who want to support
> 3.2 and 3.3 and 2.6+ in the same codebase.
> 
> "For Python 3.1 and Python 3.2 (even 3.0 if necessary) a simple 
> on-installation
> hook could be provided that tokenizes all source files and strips away the
> otherwise unnecessary u prefix at installation time."
> 
> There's some confusion about this hook - The PEP calls it an on-installation
> hook (like 2to3) but Nick said it was an import-time hook. I'm more 
> comfortable
> with the latter - it has a chance of providing an acceptable performance for a
> large codebase, as it will only kick in when .py files are newer than their
> .pyc. A 2to3 like hook, when working with a large codebase like Django, is
> likely to be about as painful as people are finding 2to3 now (when used in an
> edit-test-edit-test workflow).
> 
> "Possible Downsides" does not mention any possible adverse impact on single
> codebase for 3.2/3.3, which I mention only because it's still not clear how 
> the
> hook which is to make 3.2 development easier will work (in terms of its impact
> on development workflow).
> 
> In the section on "Modernizi

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

* Serhiy Storchaka wrote:

> 01.03.12 16:47, André Malo написав(ла):
> > On Thursday 01 March 2012 15:17:35 Serhiy Storchaka wrote:
> >> This is the first rational use of frozendict that I see. However, a
> >> deep copy is still necessary to create the frozendict. For this case,
> >> I believe, would be better to "freeze" dict inplace and then
> >> copy-on-write it.
> >
> > In my case it's actually a half one. The data mostly comes from
> > memcache ;) I'm populating the object and then I'm done with it. People
> > wanting to modify it, need to copy it, yes. OTOH usually a shallow copy
> > is enough (here).
>
> What if people modify dicts in deep?

that's the "here" part. They can't [1]. These objects are typically ROLists 
of RODicts. Maybe nested deeper, but all RO* or other immutable types.

I cheated, by deepcopying always in the cache, but defining __deepcopy__ for 
those RO* objects as "return self".

nd

[1] Well, an attacker could, because it's still based on regular dicts and 
lists. But thatswhy it's not a security feature, but a safety net (here).
-- 
"Solides und umfangreiches Buch"
  -- aus einer Rezension


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[André Malo]

* Guido van Rossum wrote:

> On Thu, Mar 1, 2012 at 9:44 AM, Victor Stinner  
wrote:
> > frozendict would help pysandbox but also any security Python module,
> > not security, but also (many) other use cases ;-)
>
> Well, let's focus on the other use cases, because to me the sandbox
> use case is too controversial (never mind how confident you are :-).
>
> I like thinking through the cache use case a bit more, since this is a
> common pattern. But I think it would be sufficient there to prevent
> accidental modification, so it should be sufficient to have a dict
> subclass that overrides the various mutating methods: __setitem__,
> __delitem__, pop(), popitem(), clear(), setdefault(), update().

For the caching part, simply making the dictproxy type public would already 
help a lot.

> What other use cases are there?

dicts as keys or as set members. I do run into this from time to time and 
always get tuple(sorted(items()) or something like that.

nd
-- 
s  s^saoaaaoaaoaaaom  a  alataa  aaoat  a  a
a maoaa a laoata  a  oia a o  a m a  o  alaoooat aaool aaoaa
matooololaaatoto  aaa o a  o ms;s;\s;s;g;y;s;:;s;y#mailto: #
 \51/\134\137| http://www.perlig.de #;print;# > n...@perlig.de
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Armin Ronacher]

Hi,

On 2/29/12 12:30 PM, Yury Selivanov wrote:
> I see you've (or somebody) changed:
Yes, I reworded that.

> Could you just remove the statement completely?
I will let Nick handle the PEP wording.

> I don't think that PEPs are the right place to put such polemic 
> and biased statements.
Why call it polemic?  If you want to use ubuntu LTS you're forcing
yourself to stick to a particular Python version for a longer time.
Which means you don't want to have to adjust your code.  Which again
means that you're better of with the Python 2.x ecosystem which is
proven, does not change nearly as quickly as the Python 3 one
(hopefully) so if you have the choice between those two you would chose
2.x over 3.x.  That's what this sentence is supposed to say.  That's not
polemic, that's just a fact.

> Nobody asked you to express your *personal* feelings and thoughts
> about applicability or state of python3 in the PEP.
That is not a personal-feeling-PEP.  If people would be 100% happy with
Python 3 we would not have these discussions, would we.

Why is it that I'm getting "attacked" on this mailinglist for writing
this PEP, or the wording etc.


Regards,
Armin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[R. David Murray]

On Thu, 01 Mar 2012 21:12:48 +, Armin Ronacher 
 wrote:
> Hi,
> 
> On 2/29/12 12:30 PM, Yury Selivanov wrote:
> > I see you've (or somebody) changed:
> Yes, I reworded that.
> 
> > Could you just remove the statement completely?
> I will let Nick handle the PEP wording.
> 
> > I don't think that PEPs are the right place to put such polemic 
> > and biased statements.
> Why call it polemic?  If you want to use ubuntu LTS you're forcing

Presumably because it comes across to him that way.

Perception aside, I do think it matches the dictionary meaning of the
term ("One who writes in support of one opinion, doctrine, or system,
in opposition to another"), which Nick's edits will presumably fix
(by addressing all sides of the argument, as a finished PEP should).

> yourself to stick to a particular Python version for a longer time.
> Which means you don't want to have to adjust your code.  Which again
> means that you're better of with the Python 2.x ecosystem which is
> proven, does not change nearly as quickly as the Python 3 one
> (hopefully) so if you have the choice between those two you would chose
> 2.x over 3.x.  That's what this sentence is supposed to say.  That's not
> polemic, that's just a fact.

Wow.  I never would have guessed that from the sentence in question.
I don't think I agree with your "that means" statement either, I
can imagine other motivations for using an LTS.  But I don't think
that discussion is worth getting in to or matters for the PEP.

> > Nobody asked you to express your *personal* feelings and thoughts
> > about applicability or state of python3 in the PEP.
> That is not a personal-feeling-PEP.  If people would be 100% happy with
> Python 3 we would not have these discussions, would we.
> 
> Why is it that I'm getting "attacked" on this mailinglist for writing
> this PEP, or the wording etc.

I think it is because people are *perceiving* that you are attacking
Python3 and arguing (out of your personal experience) that porting
is harder than other people (out of their personal experience) have
found it to be.  This presumably reflects the different problem
domains people are working in.

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sandboxing Python

[Victor Stinner]

> I challenge anymore to break pysandbox! I would be happy if anyone
> breaks it because it would make it more stronger.

Hum, I should give some rules for such contest:

- the C module (_sandbox) must be used
- you have to get access to a object outside the sandbox, like a real
module, or get access to a blocked resource (like the filesystem)
- the best is to be able to write into the filesystem
- you can use the interpreter ("python interpreter.py") to play with
the sandbox, but you have to be able to reproduce with a simple script
(e.g. using "python execfile.py script.py")

pysandbox works on Python 2.5, 2.6 and 2.7. It does not officially
support Python 3 yet.

Example.
---
$ python setup.py build
$ PYTHONPATH=build/lib.*/ python interpreter.py  --allow-path=/etc/issue
pysandbox 1.1
Enabled features: codecs, encodings, exit, interpreter, site, stderr,
stdin, stdout, traceback
(use --features=help to enable the help function)

Try to break the sandbox!

sandbox>>> open('/etc/issue').read()
'Ubuntu 11.10 \\n \\l\n\n'

sandbox>>> type(open('/etc/issue'))('test', 'w')
Traceback (most recent call last):
  File "", line 1, in 
TypeError: object.__new__() takes no parameters
---
You fail!

I'm interested by vulnerabilities in pysandbox using the Python
restricted module (used when _sandbox is missing), but it is not the
official mode :-) And it is more limited: you cannot read files for
example.

See also sandbox tests to get some ideas ;-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Barry Warsaw]

Hopefully, I can say the following in a constructive way.  I certainly don't
mean to attack anyone personally for their closely held beliefs, though I
might disagree with them.  And you have the right to those beliefs and to
express them in a respectful and constructive manner on this mailing list,
which I think you've done.  No criticisms there.

However, PEPs *are* official documents from the Python developer community, so
I think it's required of us to present technical issues in an honest light,
yet devoid of negative connotations which harm Python.

On Mar 01, 2012, at 09:12 PM, Armin Ronacher wrote:

>Why call it polemic?  If you want to use ubuntu LTS you're forcing
>yourself to stick to a particular Python version for a longer time.

Not just a particular Python 3 version, but a particular Python 2 version
too.  And a particular kernel version, and version of Perl, Ruby, Java, gcc,
etc. etc.  That's kind of the whole point of an LTS. :)

>Which means you don't want to have to adjust your code.  Which again
>means that you're better of with the Python 2.x ecosystem which is
>proven, does not change nearly as quickly as the Python 3 one
>(hopefully) so if you have the choice between those two you would chose
>2.x over 3.x.  That's what this sentence is supposed to say.  That's not
>polemic, that's just a fact.

I don't agree with the conclusion.  But none of that is germane to the PEP
anyway.

The PEP could simply say that for some domains, the ability to port code from
Python 2 to Python 3 would be enhanced by the reintroduction of the u-prefix.
It could even explain why WSGI applications in particular would benefit from
this.  That would be enough to justify Guido's acceptance of the PEP.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Yury Selivanov]

Hi Armin,

Sorry if I sounded like 'attacking' you.  I certainly had no such
intention, as I believe nobody on this list.

But if you'd just stuck to the point, without touching very 
controversial  topics of what version of python is a good choice 
and what is a bad, with full review of all porting scenarios with 
well-thought set of benchmarks, nobody would ever call your PEP 
"polemic".

Thanks,
-
Yury

On 2012-03-01, at 4:12 PM, Armin Ronacher wrote:

> Hi,
> 
> On 2/29/12 12:30 PM, Yury Selivanov wrote:
>> I see you've (or somebody) changed:
> Yes, I reworded that.
> 
>> Could you just remove the statement completely?
> I will let Nick handle the PEP wording.
> 
>> I don't think that PEPs are the right place to put such polemic 
>> and biased statements.
> Why call it polemic?  If you want to use ubuntu LTS you're forcing
> yourself to stick to a particular Python version for a longer time.
> Which means you don't want to have to adjust your code.  Which again
> means that you're better of with the Python 2.x ecosystem which is
> proven, does not change nearly as quickly as the Python 3 one
> (hopefully) so if you have the choice between those two you would chose
> 2.x over 3.x.  That's what this sentence is supposed to say.  That's not
> polemic, that's just a fact.
> 
>> Nobody asked you to express your *personal* feelings and thoughts
>> about applicability or state of python3 in the PEP.
> That is not a personal-feeling-PEP.  If people would be 100% happy with
> Python 3 we would not have these discussions, would we.
> 
> Why is it that I'm getting "attacked" on this mailinglist for writing
> this PEP, or the wording etc.
> 
> 
> Regards,
> Armin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Armin Ronacher]

Hi,

On 3/1/12 10:38 PM, Yury Selivanov wrote:
> Sorry if I sounded like 'attacking' you.  I certainly had no such
> intention, as I believe nobody on this list.
Sorry if I sound cranky but I got that impression from the responses
here (which are greatly different from the responses I got on other
communication channels and by peers).  You were just the unlucky mail I
responded to :-)

> But if you'd just stuck to the point, without touching very 
> controversial  topics of what version of python is a good choice 
> and what is a bad, with full review of all porting scenarios with 
> well-thought set of benchmarks, nobody would ever call your PEP 
> "polemic".
I tried my best but obviously it was not good enough to please
everybody.  In all honesty I did not expect that such a small change
would spawn such a great discussion.  After all what we're discussing
here is the introduction of one letter to literals :-)


Regards,
Armin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Guido van Rossum]

On Thu, Mar 1, 2012 at 12:35 PM, André Malo  wrote:
> * Guido van Rossum wrote:
>
>> On Thu, Mar 1, 2012 at 9:44 AM, Victor Stinner 
> wrote:
>> > frozendict would help pysandbox but also any security Python module,
>> > not security, but also (many) other use cases ;-)
>>
>> Well, let's focus on the other use cases, because to me the sandbox
>> use case is too controversial (never mind how confident you are :-).
>>
>> I like thinking through the cache use case a bit more, since this is a
>> common pattern. But I think it would be sufficient there to prevent
>> accidental modification, so it should be sufficient to have a dict
>> subclass that overrides the various mutating methods: __setitem__,
>> __delitem__, pop(), popitem(), clear(), setdefault(), update().
>
> For the caching part, simply making the dictproxy type public would already
> help a lot.

Heh, that's a great idea. Can you file a bug for that?

>> What other use cases are there?
>
> dicts as keys or as set members. I do run into this from time to time and
> always get tuple(sorted(items()) or something like that.

I know I've done that once or twice in my life too, but it's a pretty
rare use case and as you say the solution is simple enough. An
alternative is frozenset(d.items()) -- someone should compare the
timing of these for large dicts.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Yury Selivanov]

On 2012-03-01, at 5:52 PM, Armin Ronacher wrote:

> Hi,
> 
> On 3/1/12 10:38 PM, Yury Selivanov wrote:
>> Sorry if I sounded like 'attacking' you.  I certainly had no such
>> intention, as I believe nobody on this list.
> Sorry if I sound cranky but I got that impression from the responses
> here (which are greatly different from the responses I got on other
> communication channels and by peers).  You were just the unlucky mail I
> responded to :-)

It's OK ;)

>> But if you'd just stuck to the point, without touching very 
>> controversial  topics of what version of python is a good choice 
>> and what is a bad, with full review of all porting scenarios with 
>> well-thought set of benchmarks, nobody would ever call your PEP 
>> "polemic".
> I tried my best but obviously it was not good enough to please
> everybody.  In all honesty I did not expect that such a small change
> would spawn such a great discussion.  After all what we're discussing
> here is the introduction of one letter to literals :-)


Well, unfortunately it's not that simple from the standpoint of how
this change will be perceived by the community.  If we have u'' syntax
in python 3, will people even understand what is the key difference from 
python 2?  Will the internet be polluted with weird source-code targeted 
only for python3, but with the wide use of u''?  When to deprecate it,
and will it ever be deprecated (as everybody is already tired with all
this)?  Will it further strengthen the common misbelief the porting is
hard (as for the many of the projects it is not), or that the right way
it to have one code-base for all versions?  

And that's just the beginning of such questions.  And when this PEP was
suddenly approved, many of us felt that all those questions are not
answered and were not even discussed.

-
Yury
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Vinay Sajip]

Armin Ronacher  active-4.com> writes:

> I tried my best but obviously it was not good enough to please
> everybody.  In all honesty I did not expect that such a small change
> would spawn such a great discussion.  After all what we're discussing
> here is the introduction of one letter to literals 

The objections are not to the introduction of one letter to literals. It is the
extent to which, in your presentation of the PEP, a narrow set of concerns and a
specific addressing of those concerns has been represented as if it is the only
possible view of all right-thinking people in the Python community. What is
"obvious" to you may not be so to others - au contraire. A PEP author obviously
will promote their specific views - it is an instrument of advocacy - but an
author should not, in my view, arrogate to themselves the presumption of
speaking for everyone in the community; rather, they should respect that others
may have different sensibilities.

The PEP comes across as being primarily motivated by WSGI concerns: Nick
mentioned that he would update the PEP to "name drop" and indicate support from
you, Jacob Kaplan-Moss and Chris McDonough - all authors of Web frameworks.
While I completely acknowledge the importance and ubiquity of these web
frameworks in the overall ecosystem, I think Python the language is about more
than just Web development. There is a bit of a sense of the tail wagging the
dog. Let's remember, it's possible to do Web development without the concept of
"native" strings - this doesn't exist AFAIK in many other languages which allow
Web applications to be developed - the concept is a sort of historical accident
arising in part out of how the WSGI spec was written and evolved, interacting
with how 3.x differs from 2.x, and how some legacy APIs expect native strings
because they are broken.

There are a number of possible ways of addressing the concerns which motivated
the PEP, but in my view you have given some of them short shrift because of what
come across as personal prejudices. An example - on a Reddit thread about PEP
414, I commented:

"The PEP does not (for example) consider the possibility of leaving literals as
they are and using a n('xxx') callable for native strings. Since there are very
few places where native strings are needed, this approach is potentially less
obtrusive than either u'xxx' or u('xxx')."

Your response to this was:

"Because in all honesty, because string wrappers make a codebase horrible to
work with. I will have to continue maintaining 2.x versions for at least another
four or five years. The idea if having to use string wrappers for that long
makes me puke."

I know that this was just a comment on Reddit and was not in the PEP, but it
smacks of you throwing all your toys out of the pram. It certainly wasn't a
reasoned response to my point. And some of that toys-pram attitude bleeds
through into the language of the PEP, leading others to make the criticisms that
they have. A PEP is supposed to be balanced, reasonable and thought through.
It's not supposed to gloss over things in a hand-wavy sort of way - there's
still uncertainty in my mind, for example, whether the 3.2 hook will be a
2to3-style tool that runs over a chunk of the whole project's codebase between
editing and running a test, or whether it's an import-time hook which only kicks
in on files that have just been edited in a development session. Which of these
it is might affect crucially the experience of someone wanting to support 3.2
and 3.3 and 2.6+ - but that clearly isn't you, and you don't seem to have much
sympathy or patience with that constituency - we're all stick-in-the-muds who
want to work with Ubuntu LTS, rather than people subject to constraints imposed
by employers, clients, projects we depend on etc.

In contrast, Nick made a more reasonable case when commenting ion my preference
for unicode_literals (on this list, not on Reddit), by reminding me about how
unicode_literals changes the semantics of string literals, and this increases
the cognitive burden on developers.

I'm not whinging about the PEP in this post - I've said elsewhere that I wasn't
opposed to it. I'm just trying to respond to your apparent bewilderment at some
of the reaction to the PEP.

I have confidence that with your continued input and Nick's input, the wording
of the PEP can be made such that it doesn't ruffle quite so many feathers. I'm
looking forward to seeing the updates.

Regards,

Vinay Sajip

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Mark Janssen]

On Thu, Mar 1, 2012 at 10:00 AM, Guido van Rossum  wrote:
>
> I do know that I don't feel comfortable having a sandbox in the Python
> standard library or even recommending a 3rd party sandboxing solution
> -- if someone uses the sandbox to protect a critical resource, and a
> hacker breaks out of the sandbox, the author of the sandbox may be
> held responsible for more than they bargained for when they made it
> open source. (Doesn't an open source license limit your
> responsibility? Who knows. AFAIK this question has not gotten to court
> yet. I wouldn't want to have to go to court over it.)
>

Since there's no way (even theoretical way) to completely secure anything
(remember the DVD protection wars?), there's no way there should be any
liability if reasonable diligence is performed to provide security where
expected (which is probably calculable to some %-age of assets protected).
  It's like putting a lock on the door of your house -- you can't expect to
be held liable is someone has a crowbar.

Open sourcing code could be said to be a disclaimer on any liability as
your letting people know that you've got nothing your trying to conceal.
 It's like a dog who plays dead:  by being totally open you're actually
more secure

mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/01/2012 05:52 PM, Armin Ronacher wrote:
> Hi,
> 
> On 3/1/12 10:38 PM, Yury Selivanov wrote:
>> Sorry if I sounded like 'attacking' you.  I certainly had no such 
>> intention, as I believe nobody on this list.
> Sorry if I sound cranky but I got that impression from the responses 
> here (which are greatly different from the responses I got on other 
> communication channels and by peers).  You were just the unlucky mail
> I responded to :-)

Several responses on the list *have* been offensive, not criticizing the
PEP on its own merits but on your (presumed) motives for introducing it.
 Such attacks are wildly off-base.

>> But if you'd just stuck to the point, without touching very 
>> controversial  topics of what version of python is a good choice and
>> what is a bad, with full review of all porting scenarios with 
>> well-thought set of benchmarks, nobody would ever call your PEP 
>> "polemic".
> I tried my best but obviously it was not good enough to please 
> everybody.  In all honesty I did not expect that such a small change 
> would spawn such a great discussion.  After all what we're discussing 
> here is the introduction of one letter to literals :-)



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9QE5AACgkQ+gerLs4ltQ6m7wCdHufuDMrplgg0+MQr4M10Y4Oy
N74AoJW5UKbUjPOdrreeTT38C9Cl2eFk
=DkBX
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Victor Stinner]

Le 01/03/2012 19:07, Guido van Rossum a écrit :

What other use cases are there?


frozendict could be used to implement "read-only" types: it is not 
possible to add or remove an attribute or set an attribute value, but 
attribute value can be a mutable object. Example of an enum with my 
type_final.patch (attached to issue #14162).


>>> class Color:
...   red=1
...   green=2
...   blue=3
...   __final__=True
...
>>> Color.red
1
>>> Color.red=2
TypeError: 'frozendict' object does not support item assignment
>>> Color.yellow=4
TypeError: 'frozendict' object does not support item assignment
>>> Color.__dict__
frozendict({...})

The implementation avoids the private PyDictProxy for read-only types, 
type.__dict__ gives directly access to the frozendict (but 
type.__dict__=newdict is still blocked).


The "__final__=True" API is just a proposition, it can be anything else, 
maybe a metaclass.


Using a frozendict for type.__dict__ is not the only possible solution 
to implement read-only types. There are also Python implementation using 
properties. Using a frozendict is faster than using properties because 
getting an attribute is just a fast dictionary lookup, whereas reading a 
property requires to execute a Python function. The syntax to declare a 
read-only class is also more classic using the frozendict approach.


Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Guido van Rossum]

On Thu, Mar 1, 2012 at 4:39 PM, Victor Stinner  wrote:
> Le 01/03/2012 19:07, Guido van Rossum a écrit :
>
>> What other use cases are there?
>
>
> frozendict could be used to implement "read-only" types: it is not possible
> to add or remove an attribute or set an attribute value, but attribute value
> can be a mutable object. Example of an enum with my type_final.patch
> (attached to issue #14162).
>
 class Color:
> ...   red=1
> ...   green=2
> ...   blue=3
> ...   __final__=True
> ...
 Color.red
> 1
 Color.red=2
>
> TypeError: 'frozendict' object does not support item assignment
 Color.yellow=4
>
> TypeError: 'frozendict' object does not support item assignment
 Color.__dict__
> frozendict({...})
>
> The implementation avoids the private PyDictProxy for read-only types,
> type.__dict__ gives directly access to the frozendict (but
> type.__dict__=newdict is still blocked).
>
> The "__final__=True" API is just a proposition, it can be anything else,
> maybe a metaclass.
>
> Using a frozendict for type.__dict__ is not the only possible solution to
> implement read-only types. There are also Python implementation using
> properties. Using a frozendict is faster than using properties because
> getting an attribute is just a fast dictionary lookup, whereas reading a
> property requires to execute a Python function. The syntax to declare a
> read-only class is also more classic using the frozendict approach.

I think you should provide stronger arguments in each case why the
data needs to be truly immutable or read-only, rather than just using
a convention or an "advisory" API (like __private can be circumvented
but clearly indicates intent to the reader).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[R. David Murray]

On Thu, 01 Mar 2012 16:50:06 -0800, Guido van Rossum  wrote:
> On Thu, Mar 1, 2012 at 4:39 PM, Victor Stinner  
> wrote:
> > frozendict could be used to implement "read-only" types: it is not possible
> > to add or remove an attribute or set an attribute value, but attribute value
> > can be a mutable object. Example of an enum with my type_final.patch
> > (attached to issue #14162).
[...]
> 
> I think you should provide stronger arguments in each case why the
> data needs to be truly immutable or read-only, rather than just using
> a convention or an "advisory" API (like __private can be circumvented
> but clearly indicates intent to the reader).

+1.  Except in very limited circumstances (such as a security sandbox)
I would *much* rather have the code I'm interacting with use advisory
means rather than preventing me from being a consenting adult.  (Having to
name mangle by hand when someone has used a __ method is painful enough,
thank you...good thing the need to do that doesn't dome up often (mostly
only in unit tests)).

--David
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Nick Coghlan]

On Fri, Mar 2, 2012 at 11:50 AM, R. David Murray  wrote:
> +1.  Except in very limited circumstances (such as a security sandbox)
> I would *much* rather have the code I'm interacting with use advisory
> means rather than preventing me from being a consenting adult.  (Having to
> name mangle by hand when someone has used a __ method is painful enough,
> thank you...good thing the need to do that doesn't dome up often (mostly
> only in unit tests)).

The main argument I'm aware of in favour of this kind of enforcement
is that it means you get exceptions at the point of *error* (trying to
modify the "read-only" dict), rather than having a strange
action-at-a-distance data mutation bug to track down.

However, in that case, it's just fine (and in fact better) if there is
a way around the default enforcement via a more verbose spelling.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Yury Selivanov]

On 2012-03-01, at 7:50 PM, Guido van Rossum wrote:
> I think you should provide stronger arguments in each case why the
> data needs to be truly immutable or read-only, rather than just using
> a convention or an "advisory" API (like __private can be circumvented
> but clearly indicates intent to the reader).


Here's one more argument to support frozendicts.

For last several months I've been thinking about prohibiting coroutines
(generators + greenlets in our framework) to modify the global state.
If there is a guarantee that all coroutines of the whole application, 
modules and framework are 100% safe from that, it's possible to do some 
interesting stuff.  For instance, dynamically balance jobs across all 
application processes:

@coroutine
def on_generate_report(context):
data = yield fetch_financial_data(context)
...

In the above example, 'fetch_financial_data' may be executed in the
different process, or even on the different server, if the coroutines'
scheduler of current process decides so (based on its load, or a low 
priority of the coroutine being scheduled).

With built-in frozendict it will be easier to secure modules or
functions' __globals__ that way, allowing to play with features closer
to the ones Erlang and other concurrent languages provide.

-
Yury
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Guido van Rossum]

On Thu, Mar 1, 2012 at 6:13 PM, Yury Selivanov  wrote:
> On 2012-03-01, at 7:50 PM, Guido van Rossum wrote:
>> I think you should provide stronger arguments in each case why the
>> data needs to be truly immutable or read-only, rather than just using
>> a convention or an "advisory" API (like __private can be circumvented
>> but clearly indicates intent to the reader).
>
>
> Here's one more argument to support frozendicts.
>
> For last several months I've been thinking about prohibiting coroutines
> (generators + greenlets in our framework) to modify the global state.
> If there is a guarantee that all coroutines of the whole application,
> modules and framework are 100% safe from that, it's possible to do some
> interesting stuff.  For instance, dynamically balance jobs across all
> application processes:
>
> @coroutine
> def on_generate_report(context):
>    data = yield fetch_financial_data(context)
>    ...
>
> In the above example, 'fetch_financial_data' may be executed in the
> different process, or even on the different server, if the coroutines'
> scheduler of current process decides so (based on its load, or a low
> priority of the coroutine being scheduled).
>
> With built-in frozendict it will be easier to secure modules or
> functions' __globals__ that way, allowing to play with features closer
> to the ones Erlang and other concurrent languages provide.

That sounds *very* far-fetched. You're pretty much designing a new
language variant. It's not an argument for burdening the original
language with a data type it doesn't need for itself.

You should be able to prototype what you want using an advisory
subclass (if you subclass dict and add __slots__=[] to it, it will
cost very little overhead) or using a custom extension that implements
the flavor of frozendict that works best for you -- given that you're
already using greenlets, another extension can't be a bid burden.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a frozendict builtin type

[Yury Selivanov]

On 2012-03-01, at 9:31 PM, Guido van Rossum wrote:
> That sounds *very* far-fetched. You're pretty much designing a new
> language variant. It's not an argument for burdening the original

Yeah, that's what we do ;)

> You should be able to prototype what you want using an advisory
> subclass (if you subclass dict and add __slots__=[] to it, it will
> cost very little overhead) or using a custom extension that implements
> the flavor of frozendict that works best for you -- given that you're
> already using greenlets, another extension can't be a bid burden.

I understand.  The only reason I wrote about it is to give an idea of
how frozendicts may be used besides just sandboxing.  I'm not strongly
advocating for it, though.

-
Yury

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Éric Araujo]

Hello,

Le 02/03/2012 00:15, Yury Selivanov a écrit :
> And that's just the beginning of such questions.  And when this PEP
> was suddenly approved, many of us felt that all those questions are
> not answered and were not even discussed.

Let me comment on that “suddenly”.  We joke about Guido being the
dictator for Python, but it’s actually not a joke.  The point of the PEP
process is to help Guido make an informed decision on a proposed change.
 (There are also side benefits like providing a record of design or
implementation choices, or documenting once and for all why some idea
will never be accepted, but let’s ignore them here.)

I can’t read Guido’s mind, but I think that here he pronounced somewhat
quickly because he was convinced by the arguments in the PEP, while
choosing to ignore the problems therein, knowing that they could be
fixed later.

Regards
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 414

[Nick Coghlan]

On Fri, Mar 2, 2012 at 2:08 PM, Éric Araujo  wrote:
> I can’t read Guido’s mind, but I think that here he pronounced somewhat
> quickly because he was convinced by the arguments in the PEP, while
> choosing to ignore the problems therein, knowing that they could be
> fixed later.

It's also the case that this particular point has been the subject of
debate for a *long* time. When the decision was first made to offer
the unicode_literals future import, one of the other contenders was
just to allow the u/U prefix in Python 3 and not worry about it, and
while the "purity" side carried the day back then, it was a close run
thing. While that approach turned out to work pretty well for many
users that didn't use unicode literals all that much, the web
community understandably feel like they're being actively *punished*
for doing Unicode right in Python 2. Consider: an application that
uses 8-bit strings everywhere and blows up on non-ASCII data in Python
2 has at least a fighting chance to run unmodified *and* handle
Unicode properly on Python 3. Because unicode literals are gone, a
Unicode-aware Python 2 application currently has *no* chance to run
unmodified on Python 3.

So even though the PEP doesn't currently do a good job of *presenting*
that history to new readers, it's still very much a factor in the
decision making process.

Accordingly, I'd like to ask folks not to stress too much about the
precise wording until I get a chance to update it over the weekend :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com