Re: [Python-Dev] updating ensurepip to include wheel

2015-08-05 Thread Nick Coghlan 
On 3 August 2015 at 11:06, Donald Stufft  wrote:
>
> On August 2, 2015 at 8:47:46 PM, Robert Collins ([email protected]) 
> wrote:
>> So, pip 7.0 depends on the wheel module for its automatic wheel
>> building, and installing pip from get-pip.py, or the bundled copy in
>> virtualenvs will automatically install wheel.
>>
>> But ensurepip doesn't bundle wheel, so we're actually installing a
>> slightly crippled pip 7.1, which will lead to folk having a poorer
>> experience.
>>
>> Is this a simple bug, or do we need to update the PEP?
>>
>
> Personally, I think it's not going to be worth the pain to add wheel to
> ensurepip. We (pip) already have a somewhat rocky relationship with some
> downstream vendors because of the bundling of pip and setuptools that I'm not
> sure that wheel makes sense. Especially given that I want the optional
> dependency on Wheel to be a temporary measure until we can just implicitly
> install wheel as a build time dependency within pip and no longer need to
> install it implicitly in get-pip.py or virtualenv. In the future I expect
> setuptools to be removed as well at a similar time when we can implicitly
> install setuptools as a build time dependency of an sdist and do not require
> end users to install it explicitly.
>
> That being said, I think the PEP would need to be updated (and possibly a new
> PEP?) since we explicitly called out the fact that setuptools would currently
> be included until pip no longer needed it to be installed seperately.

I'm going to contradict what I said to Robert at the PyCon AU sprints
earlier this week, and agree with Donald here.

setuptools is in the situation where because it also includes
pkg_resources, it blurs the line between "build time" and "run time"
dependency. While it would be nice to split that and have a "just
pkg_resources" runtime dependency distinct from the build time
dependency, that isn't likely to happen any time soon.

wheel, by contrast, is already a pure build time dependency for
bdist_wheel, and thus should be getting brought in as an implied
"build requires" by pip itself when building from source. This does
pose an interesting challenge from the perspective of the "offline
installation" use case for ensurepip, where wheels are used as a local
build caching mechanism, but we don't assume PyPI access, but it isn't
one we really considered in the original ensurepip PEP.

So actually doing this would probably require a PEP to update
ensurepip with some additional options related to whether the build
dependencies should be installed or not, and give downstream vendors a
free pass to exclude the build dependencies from the default
installation set.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] updating ensurepip to include wheel

2015-08-05 Thread Donald Stufft 
On August 5, 2015 at 10:01:50 AM, Nick Coghlan ([email protected]) wrote:
>
> setuptools is in the situation where because it also includes
> pkg_resources, it blurs the line between "build time" and "run time"
> dependency. While it would be nice to split that and have a "just
> pkg_resources" runtime dependency distinct from the build time
> dependency, that isn't likely to happen any time soon.
>  
> wheel, by contrast, is already a pure build time dependency for
> bdist_wheel, and thus should be getting brought in as an implied
> "build requires" by pip itself when building from source. This does
> pose an interesting challenge from the perspective of the "offline
> installation" use case for ensurepip, where wheels are used as a local
> build caching mechanism, but we don't assume PyPI access, but it isn't
> one we really considered in the original ensurepip PEP.
> 

Just a small correction, in general setuptools does blur that line, but
for pip itself setuptools is completely a build time dependency which
isn’t *technically* any different than our dependency on wheel. We work
perfectly fine without it installed you just don’t get certain features
available to you if you don’t have it installed. However we left
setuptools installing because the feature you lose if you don’t have it
pre-installed is the ability to install from sdists entirely. It was
determined that not being able to install from sdists was a large enough
“breakage” that considering setuptools a dependency of pip in the terms
of ensurepip was considered better than minimizing the things we bundled.

On the flip side, the thing you lose if you don’t have wheel installed is
more like a “nice to have” than something that breaks functionality that
most people would consider mandatory in the current landscape.

---  
Donald Stufft  
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] updating ensurepip to include wheel

2015-08-05 Thread Nick Coghlan 
On 6 August 2015 at 00:10, Donald Stufft  wrote:
> Just a small correction, in general setuptools does blur that line, but
> for pip itself setuptools is completely a build time dependency which
> isn’t *technically* any different than our dependency on wheel. We work
> perfectly fine without it installed you just don’t get certain features
> available to you if you don’t have it installed. However we left
> setuptools installing because the feature you lose if you don’t have it
> pre-installed is the ability to install from sdists entirely. It was
> determined that not being able to install from sdists was a large enough
> “breakage” that considering setuptools a dependency of pip in the terms
> of ensurepip was considered better than minimizing the things we bundled.

Sorry, I omitted a downstream-related step in my thought process
there. We currently have a hard dependency from python to pip and
setuptools in Fedora, so we can ensure ensurepip works properly inside
virtual environments.

Enough things require setuptools at runtime for pkg_resources that
that falls into the category of "annoying runtime dependency we'd like
to see go away, but we can live with it since a lot of production
systems are still going to end up with it installed regardless of
what's in the base image".

A hard dependency on wheel wouldn't fit into the same category - when
folks are using a build pipeline to minimise the installation
footprint on production systems, the wheel package itself has no
business being installed anywhere other than developer systems and
build servers.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] who must makes FOR loop quicker

2015-08-05 Thread John Doe 
To pass by reference or by copy of - that is the question from hamlet. 
("hamlet" - a community of people smaller than a village python3.4-linux64)


xlist = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
i = 0
for x in xlist:
print(xlist)
print("\txlist[%d] = %d" % (i, x))
if x%2 == 0 :
xlist.remove(x)
print(xlist, "\n\n")
i = i + 1

So, catch the output and help me, PLEASE, improve the answer:
Does it appropriate ALWAYS REevaluate the terms of the expression list 
in FOR-scope on each iteration?
But if I want to pass ONCE a copy to FOR instead of a reference (as seen 
from an output) and reduce unreasonable reevaluation, what I must to do 
for that?

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] who must makes FOR loop quicker

2015-08-05 Thread Chris Angelico 
On Thu, Aug 6, 2015 at 1:25 AM, John Doe  wrote:
> To pass by reference or by copy of - that is the question from hamlet.
> ("hamlet" - a community of people smaller than a village python3.4-linux64)
>
> xlist = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> i = 0
> for x in xlist:
> print(xlist)
> print("\txlist[%d] = %d" % (i, x))
> if x%2 == 0 :
> xlist.remove(x)
> print(xlist, "\n\n")
> i = i + 1
>
> So, catch the output and help me, PLEASE, improve the answer:
> Does it appropriate ALWAYS REevaluate the terms of the expression list in
> FOR-scope on each iteration?
> But if I want to pass ONCE a copy to FOR instead of a reference (as seen
> from an output) and reduce unreasonable reevaluation, what I must to do for
> that?

This list is for the development *of* Python, rather than development
*with* Python. If you repost your question to [email protected]
(the main user list), I'll be happy to explain over there what's going
on and how to sort this out! But the simple answer is: Don't mutate
the thing you're iterating over. You can take a copy with xlist[:] and
iterate over that, if you like.

ChrisA
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] who must makes FOR loop quicker

2015-08-05 Thread Joe Jevnik 
The iterator is not revaluated, instead, it is constructing a single
iterator, in this case a list_iterator. The list_iterator looks at the
underyling list to know how to iterate so when you mutate the underlying
list, the list_iterator sees that. This does not mee the expression used to
generate the iterator was re-evaluated.

On Wed, Aug 5, 2015 at 11:25 AM, John Doe  wrote:

> To pass by reference or by copy of - that is the question from hamlet.
> ("hamlet" - a community of people smaller than a village python3.4-linux64)
>
> xlist = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> i = 0
> for x in xlist:
> print(xlist)
> print("\txlist[%d] = %d" % (i, x))
> if x%2 == 0 :
> xlist.remove(x)
> print(xlist, "\n\n")
> i = i + 1
>
> So, catch the output and help me, PLEASE, improve the answer:
> Does it appropriate ALWAYS REevaluate the terms of the expression list in
> FOR-scope on each iteration?
> But if I want to pass ONCE a copy to FOR instead of a reference (as seen
> from an output) and reduce unreasonable reevaluation, what I must to do for
> that?
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] who must makes FOR loop quicker

2015-08-05 Thread Steven D'Aprano 
On Wed, Aug 05, 2015 at 06:25:07PM +0300, John Doe wrote:
> To pass by reference or by copy of - that is the question from hamlet. 
> ("hamlet" - a community of people smaller than a village python3.4-linux64)
[snip question]

John, you have already posted this same question to the tutor list, 
where you have been given an answer. If the response doesn't answer your 
question, please discuss it there on the tutor list. This question is 
not suitable for this list.


-- 
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

2015-08-05 Thread Ronald Oussoren 

> On 26 Jul 2015, at 14:18, Mark Shannon  wrote:
> 
>> On 26 July 2015 at 10:41 Ronald Oussoren  wrote:
>> 
>> 
>> 
>>> On 26 Jul 2015, at 09:14, Ronald Oussoren  wrote:
>>> 
>>> 
 On 25 Jul 2015, at 17:39, Mark Shannon >>> > wrote:
 
 Hi,
 
 On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
> 
> Another summer with another EuroPython, which means its time again to 
> try to revive PEP 447…
> 
 
 IMO, there are two main issues with the PEP and implementation.
 
 1. The implementation as outlined in the PEP is infinitely recursive, since
 the
 lookup of "__getdescriptor__" on type must necessarily call
 type.__getdescriptor__.
 The implementation (in C) special cases classes that inherit
 "__getdescriptor__"
 from type. This special casing should be mentioned in the PEP.
>>> 
>>> Sure.  An alternative is to slightly change the the PEP: use
>>> __getdescriptor__ when
>>> present and directly peek into __dict__ when it is not, and then remove the
>>> default
>>> __getdescriptor__. 
>>> 
>>> The reason I didn’t do this in the PEP is that I prefer a programming model
>>> where
>>> I can explicitly call the default behaviour. 
>> 
>> I’m not sure there is a problem after all (but am willing to use the
>> alternative I describe above),
>> although that might be because I’m too much focussed on CPython semantics.
>> 
>> The __getdescriptor__ method is a slot in the type object and because of that
>> the
>> normal attribute lookup mechanism is side-stepped for methods implemented in
>> C. A
>> __getdescriptor__ that is implemented on Python is looked up the normal way 
>> by
>> the 
>> C function that gets added to the type struct for such methods, but that’s 
>> not
>> a problem for
>> type itself.
>> 
>> That’s not new for __getdescriptor__ but happens for most other special
>> methods as well,
>> as I noted in my previous mail, and also happens for the __dict__ lookup
>> that’s currently
>> used (t.__dict__ is an attribute and should be lookup up using
>> __getattribute__, …)
> 
> 
> "__getdescriptor__" is fundamentally different from "__getattribute__" in that
> is defined in terms of itself.
> 
> object.__getattribute__ is defined in terms of type.__getattribute__, but
> type.__getattribute__ just does 
> dictionary lookups.

object.__getattribute__ is actually defined in terms of type.__dict__ and 
object.__dict__. Type.__getattribute__ is at best used to to find type.__dict__.

> However defining type.__getattribute__ in terms of
> __descriptor__ causes a circularity as
> __descriptor__ has to be looked up on a type.
> 
> So, not only must the cycle be broken by special casing "type", but that
> "__getdescriptor__" can be defined
> not only by a subclass, but also a metaclass that uses "__getdescriptor__" to
> define  "__getdescriptor__" on the class.
> (and so on for meta-meta classes, etc.)

Are the semantics of special methods backed by a member in PyTypeObject part of 
Python’s semantics, or are those CPython implementation details/warts? In 
particular that such methods are access directly without using __getattribute__ 
at all (or at least only indirectly when the method is implemented in Python).  
That is:

>>> class Dict (dict):
...def __getattribute__(self, nm):
...   print("Get", nm)
...   return dict.__getattribute__(self, nm)
... 
>>> 
>>> d = Dict(a=4)
>>> d.__getitem__('a')
Get __getitem__
4
>>> d['a']
4
>>> 

(And likewise for other special methods, which amongst others means that 
neither __getattribute__ nor __getdescriptor__ can be used to dynamicly add 
such methods to a class)

In my proposed patch I do special case “type”, but that’s only intended as a 
(for now unbenchmarked) speed hack.  The code would work just as well without 
the hack because the metatype’s  __getdescriptor__ is looked up directly in the 
PyTypeObject on the C level, without using __getattribute__ and hence without 
having to use recursion.

BTW. I wouldn’t mind dropping the default “type.__getdescriptor__” completely 
and reword my proposal to state that __getdescriptor__ is used when present, 
and otherwise __dict__ is accessed directly.  That would remove the infinite 
recursion, as all metaclass chains should at some end up at “type” which then 
wouldn’t have a “__getdescriptor__”.   

The reason I added “type.__getdescriptor__” is that IMHO gives a nicer 
programming model where you can call the superclass implementation in the 
implementation of __getdescriptor__ in a subclass.  Given the minimal semantics 
of “type.__getdescriptor__”  loosing that wouldn’t be too bad to get a better 
object model.

Ronald

> 
> Cheers,
> Mark
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

___

Re: [Python-Dev] updating ensurepip to include wheel

2015-08-05 Thread Victor Stinner 
Le 5 août 2015 17:12, "Nick Coghlan"  a écrit :
> A hard dependency on wheel wouldn't fit into the same category - when
> folks are using a build pipeline to minimise the installation
> footprint on production systems, the wheel package itself has no
> business being installed anywhere other than developer systems and
> build servers.

I'm quite sure that virtualenv is used to deploy python on production.

Pip 7 automatically creates wheel packages when no build wheel package is
available on PyPI. Examples numpy and any pure python package only
providing a tarball.

For me it makes sense to embed wheel in ensurepip and to install wheel on
production systems (to install pacakes, not to build them).

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] updating ensurepip to include wheel

2015-08-05 Thread Nick Coghlan 
On 6 August 2015 at 09:29, Victor Stinner  wrote:
> Le 5 août 2015 17:12, "Nick Coghlan"  a écrit :
>> A hard dependency on wheel wouldn't fit into the same category - when
>> folks are using a build pipeline to minimise the installation
>> footprint on production systems, the wheel package itself has no
>> business being installed anywhere other than developer systems and
>> build servers.
>
> I'm quite sure that virtualenv is used to deploy python on production.
>
> Pip 7 automatically creates wheel packages when no build wheel package is
> available on PyPI. Examples numpy and any pure python package only providing
> a tarball.
>
> For me it makes sense to embed wheel in ensurepip and to install wheel on
> production systems (to install pacakes, not to build them).

pip can install from wheels just fine without the wheel package being
present - that's how ensurepip already works.

The wheel package itself is only needed in order to support the
setuptools "bdist_wheel" command, which then allows pip to implicitly
cache wheel files when installing from an sdist.

Installing from sdist in production is a *fundamentally bad idea*,
because it means you have to have a build toolchain on your production
servers. One of the benefits of the wheel format and projects like
devpi is that it makes it easier to discourage people from doing that.
Even without getting into Linux containers and tools like pyp2rpm,
it's also possible to create an entire virtualenv on a build server,
bundle that up as an RPM or DEB file, and use the system package
manager to do the production deployment.

However, production Linux servers aren't the only case we need to care
about, and there's a strong user experience argument to be made for
providing wheel by default upstream, and telling downstream
redistributors that care about the distinction to do the necessary
disentangling to make it easy to have "build dependency free"
production images.

We've learned from experience that things go far more smoothly if we
thrash out those kinds of platform dependent behavioural differences
*before* we inflict them on end users, rather than having downstream
redistributors tackle foreseeable problems independently of both each
other and upstream :)

Hence my request for a PEP - I can see why adding wheel to the
ensurepip bundle would be a good idea for upstream, but I can also see
why it's a near certainty downstream Linux distros (including Fedora)
would take it out again in at least some situations to better meet the
needs of *our* user base. (Since RPM has weak dependency support now,
we'd likely make python-wheel a "Recommends:" dependency, rather than
a "Requires:" dependency - still installed by default, but easy to
omit if not wanted or needed)

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com