Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Dima Tisnek]

Hey I've taken time to read the PEP, my 2c... actually 1c:

Most important is to explain how this changes behaviour of Python programs.

A comprehensive set of Python examples where behaviour is changed (for
better or worse) please.

While I understand the concern of "superclasses of objects that gain
or lose attributes at runtime" on the theoretical level, please
translate that into actual Python examples.

d.

On 22 July 2015 at 09:25, Ronald Oussoren  wrote:
> Hi,
>
> Another summer with another EuroPython, which means its time again to try to
> revive PEP 447…
>
> I’ve just pushes a minor update to the PEP and would like to get some
> feedback on this, arguably fairly esoteric, PEP.
>
> The PEP proposes to to replace direct access to the class __dict__ in
> object.__getattribute__ and super.__getattribute__ by calls to a new special
> method to give the metaclass more control over attribute lookup, especially
> for access using a super() object.  This is needed for classes that cannot
> store (all) descriptors in the class dict for some reason, see the PEP text
> for a real-world example of that.
>
> Regards,
>
>   Ronald
>
>
> The PEP text (with an outdated section with benchmarks removed):
>
> PEP: 447
> Title: Add __getdescriptor__ method to metaclass
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ronald Oussoren 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Jun-2013
> Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
>
>
> Abstract
> 
>
> Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek
> in the ``__dict__`` of classes on the MRO for a class when looking for
> an attribute. This PEP adds an optional ``__getdescriptor__`` method to
> a metaclass that replaces this behavior and gives more control over
> attribute
> lookup, especially when using a `super`_ object.
>
> That is, the MRO walking loop in ``_PyType_Lookup`` and
> ``super.__getattribute__`` gets changed from::
>
>  def lookup(mro_list, name):
>  for cls in mro_list:
>  if name in cls.__dict__:
>  return cls.__dict__
>
>  return NotFound
>
> to::
>
>  def lookup(mro_list, name):
>  for cls in mro_list:
>  try:
>  return cls.__getdescriptor__(name)
>  except AttributeError:
>  pass
>
>  return NotFound
>
> The default implemention of ``__getdescriptor__`` looks in the class
> dictionary::
>
>class type:
>   def __getdescriptor__(cls, name):
>   try:
>   return cls.__dict__[name]
>   except KeyError:
>   raise AttributeError(name) from None
>
> Rationale
> =
>
> It is currently not possible to influence how the `super class`_ looks
> up attributes (that is, ``super.__getattribute__`` unconditionally
> peeks in the class ``__dict__``), and that can be problematic for
> dynamic classes that can grow new methods on demand.
>
> The ``__getdescriptor__`` method makes it possible to dynamically add
> attributes even when looking them up using the `super class`_.
>
> The new method affects ``object.__getattribute__`` (and
> `PyObject_GenericGetAttr`_) as well for consistency and to have a single
> place to implement dynamic attribute resolution for classes.
>
> Background
> --
>
> The current behavior of ``super.__getattribute__`` causes problems for
> classes that are dynamic proxies for other (non-Python) classes or types,
> an example of which is `PyObjC`_. PyObjC creates a Python class for every
> class in the Objective-C runtime, and looks up methods in the Objective-C
> runtime when they are used. This works fine for normal access, but doesn't
> work for access with `super`_ objects. Because of this PyObjC currently
> includes a custom `super`_ that must be used with its classes, as well as
> completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute
> access.
>
> The API in this PEP makes it possible to remove the custom `super`_ and
> simplifies the implementation because the custom lookup behavior can be
> added in a central location.
>
> .. note::
>
>`PyObjC`_ cannot precalculate the contents of the class ``__dict__``
>because Objective-C classes can grow new methods at runtime. Furthermore
>Objective-C classes tend to contain a lot of methods while most Python
>code will only use a small subset of them, this makes precalculating
>unnecessarily expensive.
>
>
> The superclass attribute lookup hook
> 
>
> Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
> `PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code)
> walk an object's MRO and currently peek in the class' ``__dict__`` to look
> up
> attributes.
>
> With this proposal both lookup methods no longer peek in the class
> ``__dict__``
> but call the special method ``__getdescriptor__``, which is a slot defined
> on the metaclass.

Re: [Python-Dev] How do we tell if we're helping or hindering the core development process? (was Re: How far to go with user-friendliness)

[Paul Moore]

On 23 July 2015 at 03:01, Alexander Belopolsky
 wrote:
> On Wed, Jul 22, 2015 at 7:09 AM, Paul Moore  wrote:
>>
>> does anyone seriously think a core dev
>> commits code as a joke???
>
>
> Yes, . :-)

Second person to pick me up on that :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Ronald Oussoren]

> On 23 Jul 2015, at 11:29, Dima Tisnek  wrote:
> 
> Hey I've taken time to read the PEP, my 2c... actually 1c:
> 
> Most important is to explain how this changes behaviour of Python programs.
> 
> A comprehensive set of Python examples where behaviour is changed (for
> better or worse) please.

The behaviour of existing python code is not changed at all.  Code that 
directly looks in the class __dict__ might be impacted, but only when running 
into classes with a custom __getdescriptor__ method. I’ve listed the code in 
the stdlib that could be affected, but have to do a new pass of the stdlib to 
check if anything relevant changed since I wrote the section.   I general you 
run into the same issues when adding a custom __getattribute__ or __getattr__ 
method, both of which will confuse introspection tools that assume regular 
attribute lookup semantics.

> 
> While I understand the concern of "superclasses of objects that gain
> or lose attributes at runtime" on the theoretical level, please
> translate that into actual Python examples.

The primary use case I have for this are classes that are proxies for external 
systems. There may be other uses as well, but I don’t have examples of that 
(other than the contrived example in the PEP).

The reason I wrote the PEP in the first place is PyObjC: this project defines a 
proxy later between Python and Objective-C, with the goal to making it possible 
to write programs for Mac OS X in Python while being able to make full use of 
Apple’s high-level APIs.  The proxy layer is mostly completely dynamic: proxies 
for Objective-C classes and their methods are created at runtime by inspecting 
the Objective-C runtime with optionally extra annotations (provided by the 
project) for stuff that cannot be extracted from the runtime. 

That is, at runtime PyObjC creates a Python class “NSObject” that corresponds 
to the Objective-C class “NSObject” as defined by Apple. Every method of the 
Objective-C class is make available as a method on the Python proxy class as 
well.

It is not possible to 100% reliably set up the Python proxy class for 
“NSObject” with all methods because Objective-C classes can grow new methods at 
runtime, and the introspection API that Apple provides does not have a way to 
detect this other than by polling.  Older versions of PyObjC did poll, but even 
that was not relialble enough and left a race condition:

 def someAction_(self, sender):
   self.someMethod()
   self.button.setTitle_(“click me”)
   super().someOtherMethod()

The call to “someMethod” used to poll the Objective-C runtime for changes.  The 
call through super() of someOtherMethod() does not do so because of the current 
semantics of super (which PEP 447 tries to change). That’s a problem because 
“self.button.setTitle_” might load a bundle that adds “someOtherMethod” to one 
of our super classes. That sadly enough is not a theoretical concern, I’ve seen 
something like this in the past.

Because of this PyObjC contains its own version of builtins.super which must be 
used with it (and is fully compatible with builtin.super for other classes).

Recent versions of PyObjC no longer poll, primarily because polling is costly 
and because Objective-C classes tend to have fat APIs most of which is never 
used by any one program.

What bothers me with PyObjC’s current approach is one the one hand that a 
custom super is inherently incompatible with any other library that might have 
a simular need, and on there other hand that I have to reimplement all logic in 
both object.__getattribute__ and super.__getattribute__ to be able to have a 
customisation of one small aspect of attribute lookup.

Ronald

> 
> d.
> 
> On 22 July 2015 at 09:25, Ronald Oussoren  wrote:
>> Hi,
>> 
>> Another summer with another EuroPython, which means its time again to try to
>> revive PEP 447…
>> 
>> I’ve just pushes a minor update to the PEP and would like to get some
>> feedback on this, arguably fairly esoteric, PEP.
>> 
>> The PEP proposes to to replace direct access to the class __dict__ in
>> object.__getattribute__ and super.__getattribute__ by calls to a new special
>> method to give the metaclass more control over attribute lookup, especially
>> for access using a super() object.  This is needed for classes that cannot
>> store (all) descriptors in the class dict for some reason, see the PEP text
>> for a real-world example of that.
>> 
>> Regards,
>> 
>>  Ronald
>> 
>> 
>> The PEP text (with an outdated section with benchmarks removed):
>> 
>> PEP: 447
>> Title: Add __getdescriptor__ method to metaclass
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Ronald Oussoren 
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 12-Jun-2013
>> Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
>> 
>> 
>> Abstract
>> 
>> 
>> Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek
>> in the ``__dict__`` 

Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Joao S. O. Bueno]

On 22 July 2015 at 14:21, Ronald Oussoren  wrote:
>>
>> 2. Is this useful, that you can think of, for anything other than connecting 
>> to Objective C?
>
> Not immediately.  But then again, I initially thought that decorators would 
> have limited appeal as well :-).  I guess this could be useful for other 
> proxy-like objects as well, especially when preloading the __dict__ is 
> relatively expensive.
>

I think one place this could be immediately useful is in code using
remote method invocation/remote attribute access. This allows one to
subclass proxy types for remote objects, and call methods that resolve
remotely in a seamless way.
(And without having to download and pre-populate an entire API into
the proxy-class __dict__)

+1
I found the solution rather concise for the flexibility it adds as well.
> Apart from direct usefulness this closes a hole in the way you can influence 
> attribute lookup.
>
> Ronald
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Guido van Rossum]

Sorry for reviving this thread, but I was cornered at EuroPython with a
question about the status of this PEP. It seems the proposal ran out of
steam and has now missed the Python 3.5 train. What happened? Is the
problem unsolvable? Or could we get this into 3.6???
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Steve Dower]

I wonder whether XML RPC might be a good example? After all, it's already in 
the stdlib and presumably suffers from the same issue.

Cheers,
Steve

Top-posted from my Windows Phone

From: Ronald Oussoren
Sent: ‎7/‎23/‎2015 3:07
To: Dima Tisnek
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 447 (type.__getdescriptor__)


> On 23 Jul 2015, at 11:29, Dima Tisnek  wrote:
>
> Hey I've taken time to read the PEP, my 2c... actually 1c:
>
> Most important is to explain how this changes behaviour of Python programs.
>
> A comprehensive set of Python examples where behaviour is changed (for
> better or worse) please.

The behaviour of existing python code is not changed at all.  Code that 
directly looks in the class __dict__ might be impacted, but only when running 
into classes with a custom __getdescriptor__ method. I’ve listed the code in 
the stdlib that could be affected, but have to do a new pass of the stdlib to 
check if anything relevant changed since I wrote the section.   I general you 
run into the same issues when adding a custom __getattribute__ or __getattr__ 
method, both of which will confuse introspection tools that assume regular 
attribute lookup semantics.

>
> While I understand the concern of "superclasses of objects that gain
> or lose attributes at runtime" on the theoretical level, please
> translate that into actual Python examples.

The primary use case I have for this are classes that are proxies for external 
systems. There may be other uses as well, but I don’t have examples of that 
(other than the contrived example in the PEP).

The reason I wrote the PEP in the first place is PyObjC: this project defines a 
proxy later between Python and Objective-C, with the goal to making it possible 
to write programs for Mac OS X in Python while being able to make full use of 
Apple’s high-level APIs.  The proxy layer is mostly completely dynamic: proxies 
for Objective-C classes and their methods are created at runtime by inspecting 
the Objective-C runtime with optionally extra annotations (provided by the 
project) for stuff that cannot be extracted from the runtime.

That is, at runtime PyObjC creates a Python class “NSObject” that corresponds 
to the Objective-C class “NSObject” as defined by Apple. Every method of the 
Objective-C class is make available as a method on the Python proxy class as 
well.

It is not possible to 100% reliably set up the Python proxy class for 
“NSObject” with all methods because Objective-C classes can grow new methods at 
runtime, and the introspection API that Apple provides does not have a way to 
detect this other than by polling.  Older versions of PyObjC did poll, but even 
that was not relialble enough and left a race condition:

 def someAction_(self, sender):
   self.someMethod()
   self.button.setTitle_(“click me”)
   super().someOtherMethod()

The call to “someMethod” used to poll the Objective-C runtime for changes.  The 
call through super() of someOtherMethod() does not do so because of the current 
semantics of super (which PEP 447 tries to change). That’s a problem because 
“self.button.setTitle_” might load a bundle that adds “someOtherMethod” to one 
of our super classes. That sadly enough is not a theoretical concern, I’ve seen 
something like this in the past.

Because of this PyObjC contains its own version of builtins.super which must be 
used with it (and is fully compatible with builtin.super for other classes).

Recent versions of PyObjC no longer poll, primarily because polling is costly 
and because Objective-C classes tend to have fat APIs most of which is never 
used by any one program.

What bothers me with PyObjC’s current approach is one the one hand that a 
custom super is inherently incompatible with any other library that might have 
a simular need, and on there other hand that I have to reimplement all logic in 
both object.__getattribute__ and super.__getattribute__ to be able to have a 
customisation of one small aspect of attribute lookup.

Ronald

>
> d.
>
> On 22 July 2015 at 09:25, Ronald Oussoren  wrote:
>> Hi,
>>
>> Another summer with another EuroPython, which means its time again to try to
>> revive PEP 447…
>>
>> I’ve just pushes a minor update to the PEP and would like to get some
>> feedback on this, arguably fairly esoteric, PEP.
>>
>> The PEP proposes to to replace direct access to the class __dict__ in
>> object.__getattribute__ and super.__getattribute__ by calls to a new special
>> method to give the metaclass more control over attribute lookup, especially
>> for access using a super() object.  This is needed for classes that cannot
>> store (all) descriptors in the class dict for some reason, see the PEP text
>> for a real-world example of that.
>>
>> Regards,
>>
>>  Ronald
>>
>>
>> The PEP text (with an outdated section with benchmarks removed):
>>
>> PEP: 447
>>

Re: [Python-Dev] Status on PEP-431 Timezones

[Lennart Regebro]

On Thu, Jul 23, 2015 at 5:07 PM, Guido van Rossum  wrote:
> Sorry for reviving this thread, but I was cornered at EuroPython with a
> question about the status of this PEP. It seems the proposal ran out of
> steam and has now missed the Python 3.5 train. What happened? Is the problem
> unsolvable? Or could we get this into 3.6???

It turns out it's very complex to solve this when internally storing
the time as the local time. Basically you have to normalize the time
(ie check if daylight savings have changed) when doing arithmetic, but
normalize is doing arithmetic, and you get infinite recursion. I've
tried various ways to solve this but ran out of steam/brainpower.

I think we should look into storing as UTC internally instead. It's a
big change (and also needs handling pickles in a backwards compatible
way) but I think that's the way forward.

//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: Python 3.5 beta 4 is tagged in one week

[Robert Collins]

On 22 July 2015 at 08:07, Robert Collins  wrote:
> On 22 July 2015 at 05:08, Larry Hastings  wrote:
>>
>>
>> On 07/21/2015 06:35 PM, Robert Collins wrote:
>>
>> Cool. http://bugs.python.org/issue21750 is in a bad state right now.
>>
>> I landed a patch to fix it, which when exposed to users had some
>> defects. I'm working on a better patch now, but need to either roll
>> the prior patch completely back, or get the new one landed before the
>> next beta. I hope to have that up for review later today {fingers
>> crossed} - will that be soon enough, or should I look up how to easily
>> revert stuff out with hg?
>>
>>
>> If you want to undo it, "hg backout" is the command you want.  In general
>> it's best to not check in broken stuff.
>
> Thanks. And yes, naturally - we didn't realise it was broken. Passing
> tests != fit for purpose.

21750 is now sorted out in the cpython repo.

I have a separate question for you - issue2091 has a good patch on it,
but would you like it added to 3.5?

It makes a broken combination of file modes - rU+ - a clean error, and
tweaks the existing exception text for U + writing modes.

-Rob




-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Guido van Rossum]

Can you update the PEP with a small note about this and update the status
to Postponed? Switching to UTC is a big change indeed.

Or could you leave this issue unsolved and still make progress with the tz
database?

In any case, new discussion should then go back to python-ideas.

On Thu, Jul 23, 2015 at 6:22 PM, Lennart Regebro  wrote:

> On Thu, Jul 23, 2015 at 5:07 PM, Guido van Rossum 
> wrote:
> > Sorry for reviving this thread, but I was cornered at EuroPython with a
> > question about the status of this PEP. It seems the proposal ran out of
> > steam and has now missed the Python 3.5 train. What happened? Is the
> problem
> > unsolvable? Or could we get this into 3.6???
>
> It turns out it's very complex to solve this when internally storing
> the time as the local time. Basically you have to normalize the time
> (ie check if daylight savings have changed) when doing arithmetic, but
> normalize is doing arithmetic, and you get infinite recursion. I've
> tried various ways to solve this but ran out of steam/brainpower.
>
> I think we should look into storing as UTC internally instead. It's a
> big change (and also needs handling pickles in a backwards compatible
> way) but I think that's the way forward.
>
> //Lennart
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python automatic optimization

[Andrew Steinberg via Python-Dev]

Hello everybody,
I am using Python 2.7 as a backbone for some mathematical simulations. I 
recently discovered a tool called AutoFDO and I tried compiling my own Python 
version, but I did not manage to get it working. My question is, will sometime 
in the future Python include this tool?
Thank you,Andrew___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python automatic optimization

[Maciej Fijalkowski]

As far as I can tell, the feedback directed optimizations don't give
much speedup on Python. There is a variety of tools for help: cython,
numba, pypy, numpy etc. if you care about performance of mathematical
operations.

On Thu, Jul 23, 2015 at 9:04 PM, Andrew Steinberg via Python-Dev
 wrote:
> Hello everybody,
>
> I am using Python 2.7 as a backbone for some mathematical simulations. I
> recently discovered a tool called AutoFDO and I tried compiling my own
> Python version, but I did not manage to get it working. My question is, will
> sometime in the future Python include this tool?
>
> Thank you,
> Andrew
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python automatic optimization

[Stefan Behnel]

Andrew Steinberg via Python-Dev schrieb am 23.07.2015 um 21:04:
> Hello everybody, I am using Python 2.7 as a backbone for some
> mathematical simulations. I recently discovered a tool called AutoFDO
> and I tried compiling my own Python version, but I did not manage to get
> it working. My question is, will sometime in the future Python include
> this tool? Thank you,Andrew

Some Python distributions already use PGO/FDO to build the interpreter.
Ubuntu does it, for example, and potentially also other Linux distributions.

Also see this issue:

https://bugs.python.org/issue17781

Apart from that, I concur with Maciej that FDO is most likely not the
solution to your problem at hand. But that belongs on the general Python
mailing list then, not the core developers mailing list.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Alexander Belopolsky]

On Thu, Jul 23, 2015 at 12:22 PM, Lennart Regebro  wrote:

> It turns out it's very complex to solve this when internally storing
> the time as the local time. Basically you have to normalize the time
> (ie check if daylight savings have changed) when doing arithmetic, but
> normalize is doing arithmetic, and you get infinite recursion.
>

This is not true.  Tim's analysis immortalized [1] at the end of the
datetime.py file,
shows that UTC to local mapping can be unambiguously recovered from the
local to UTC rules using a simple finite algorithm.  Tim assumes [2] that
standard (non-DST)
time offset is constant throughout the history, but this requirement can be
relaxed to offset
changing no more than once in any 48 hour period (if you generously allow
timezones
from -24 to 24 hours).

Actually, it looks like I am repeating what I wrote back in April, so I'll
stop with a
reference [3] to that post.

[1]: https://hg.python.org/cpython/file/v3.5.0b1/Lib/datetime.py#l1935
[2]: https://hg.python.org/cpython/file/v3.5.0b1/Lib/datetime.py#l1948
[3]: https://mail.python.org/pipermail/python-dev/2015-April/139171.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com