Re: [Distutils] PyPI index workaround

[Matt Bacchi]

OT: I hope you're going to provide a setting to allow the user to disable
this unnecessary and bandwith intensive 'feature'?

-Matt

From: "Михаил Голубев" 
To: Donald Stufft 
Cc: Dmitry Trofimov ,
distutils-sig@python.org
Date: Wed, 13 Jul 2016 23:21:24 +0300
Subject: Re: [Distutils] PyPI index workaround
Right, sorry, that initial question wasn't clear about that.

We need the latest versions only for installed packages. Nonetheless, as
you noted, it's still several dozens consecutive requests to
"/simple/" for each PyCharm session of every user.

Can you handle that?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

I forget the exact names but there's a range of SQL Server packages that also 
fit in here. Perhaps I get to hear more complaints about those because of where 
I work :)

But you're right, it may be a small enough problem to handle it that way.

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎7/‎14/‎2016 17:25
To: "Steve Dower" 
Cc: "Daniel D. Beck" ; "distutils-sig" 

Subject: Re: [Distutils] Outdated packages on pypi



On Jul 14, 2016, at 8:19 PM, Steve Dower  wrote:


I'm still keen to find a way to redirect people to useful forks or alternative 
packages that doesn't require thousands of mentions at conferences for all time 
ala PIL.




I’m not opposed to this but we’ll want to make sure we’re careful about how we 
do it. PIL is an easy example where the maintainer is gone and there is a 
community fork of it. But I struggle to come up with very many more examples of 
this where there is something that is:


- Popular enough that enough people are tripping over it to make it worth it.
- There is a clear successor (or successors).


Off the top of my head I can only really think of PIL, and *maybe* suds. Unless 
there’s a lot of these maybe all we really need is a policy for when 
administrators can/will edit the page to direct people towards a different 
project or a way to add an admin message directing people to another project.

—
Donald Stufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Donald Stufft]

> On Jul 14, 2016, at 8:19 PM, Steve Dower  wrote:
> 
> I'm still keen to find a way to redirect people to useful forks or 
> alternative packages that doesn't require thousands of mentions at 
> conferences for all time ala PIL.


I’m not opposed to this but we’ll want to make sure we’re careful about how we 
do it. PIL is an easy example where the maintainer is gone and there is a 
community fork of it. But I struggle to come up with very many more examples of 
this where there is something that is:

- Popular enough that enough people are tripping over it to make it worth it.
- There is a clear successor (or successors).

Off the top of my head I can only really think of PIL, and *maybe* suds. Unless 
there’s a lot of these maybe all we really need is a policy for when 
administrators can/will edit the page to direct people towards a different 
project or a way to add an admin message directing people to another project.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

Sure, if it's been tried before and people couldn't control themselves back 
then (before my time too, and the internet wasn't as blatantly toxic six+ years 
ago as it is now) then that's reason enough not to try again.

I'm still keen to find a way to redirect people to useful forks or alternative 
packages that doesn't require thousands of mentions at conferences for all time 
ala PIL.

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎7/‎14/‎2016 17:06
To: "Steve Dower" 
Cc: "Daniel D. Beck" ; "distutils-sig" 

Subject: Re: [Distutils] Outdated packages on pypi


> On Jul 14, 2016, at 6:51 PM, Steve Dower  wrote:
> 
> On 14Jul2016 0619, Daniel D. Beck wrote:
>> Free-form, user-generated content on PyPI would become a pathway for
>> harassment and abuse. Introducing user-generated content on PyPI would
>> necessarily put an emotional burden on package maintainers in addition
>> to the maintenance burden (unless PyPI moderators are going to screen
>> content before maintainers and users see it—given the dearth of
>> resources for PyPI as it is, this strikes me as exceedingly unlikely).
> 
> This is why I listed a set of restrictions to help prevent that:
> 
> * 140 chars (flexible, but short enough to prevent rants)
> * users must be logged in
> * no external links
> * maintainers can delete/dispute comments
> * clear comments on each new release
> * one comment per user per package (implied, but I didn't explicitly call it 
> out in my previous email)
> 
> Do you really think this will be worse than the current state, where abusers 
> *only* have access Twitter, github, reddit and email to harass package 
> maintainers?
> 
> Assuming harassment is not going to be a problem, is there value in letting 
> people add comments directly on the page where users seem to keep ending up?
> 

I don’t believe you can assume that harassment is not going to be a problem.

There’s a fundamental power dynamic here, where publishing your project to PyPI 
is not *entirely* optional. It’s optional in the sense that nobody is going to 
force you to publish your project there, but it’s hard to interact fully with 
the Python ecosystem as a whole for your project if you don’t at least add an 
entry for it there. Given that we have this dynamic, we need to be particularly 
careful how the features we add can be used against people, particularly 
against the most vulnerable people in our community.

We sadly live in a world where our industry is incredibly toxic to, well 
basically everyone but white guys and are actively hostile towards efforts to 
seeing a community become more inclusive. These are people who will regularly 
create multiple twitter accounts in order to spam harassment at people (in 140 
characters) to get around cases where the person has blocked them. These are 
people who will flood comments on GitHub issue trackers for projects they don’t 
even use to bitch about someone changing some pronouns to be more inclusive.

Consider that a rude comment can completely crush someone’s motivation to learn 
Python, or to maintain a package. It can make our community seem all that more 
hostile and I don’t think the vast majority of comments are going to actually 
be very useful. I suspect they will largely be used as yet another support 
venue for random users who are confused (and deleting them doesn’t help those 
users either).

We had a comments and review system years ago (before my time TBH) and the 
backlash against it was so great that it was a major point of contention on 
catalog-sig where Package authors wanted it to be gotten rid of and the 
maintainers at the time pushing back to keep it. We (obviously) eventually got 
rid of it, and I think that is pretty indicative of the idea in general.

Any sort of user created content requires us, the people running PyPI, to 
moderate to some degree. We have to do it now with people who create projects 
with vulgar or offensive names and I don’t believe that we have the man power 
available to us to moderate the comments of a much larger feature that is going 
to incentivize people to make negative comments (and let’s be real, 95% of the 
comments are going to be negative, people rarely reach out to say they’re happy 
but they’re always ready to complain).

This is a lot of words to say that I would be very against this kind of feature 
on PyPI. I am not *entirely* against some sort of automated marker for possibly 
unmaintained packages, but even that I’m sketchy on. Allowing people to poop 
their own content onto project pages for a project they don’t own is just not 
tenable I think.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Donald Stufft]

> On Jul 14, 2016, at 6:51 PM, Steve Dower  wrote:
> 
> On 14Jul2016 0619, Daniel D. Beck wrote:
>> Free-form, user-generated content on PyPI would become a pathway for
>> harassment and abuse. Introducing user-generated content on PyPI would
>> necessarily put an emotional burden on package maintainers in addition
>> to the maintenance burden (unless PyPI moderators are going to screen
>> content before maintainers and users see it—given the dearth of
>> resources for PyPI as it is, this strikes me as exceedingly unlikely).
> 
> This is why I listed a set of restrictions to help prevent that:
> 
> * 140 chars (flexible, but short enough to prevent rants)
> * users must be logged in
> * no external links
> * maintainers can delete/dispute comments
> * clear comments on each new release
> * one comment per user per package (implied, but I didn't explicitly call it 
> out in my previous email)
> 
> Do you really think this will be worse than the current state, where abusers 
> *only* have access Twitter, github, reddit and email to harass package 
> maintainers?
> 
> Assuming harassment is not going to be a problem, is there value in letting 
> people add comments directly on the page where users seem to keep ending up?
> 

I don’t believe you can assume that harassment is not going to be a problem.

There’s a fundamental power dynamic here, where publishing your project to PyPI 
is not *entirely* optional. It’s optional in the sense that nobody is going to 
force you to publish your project there, but it’s hard to interact fully with 
the Python ecosystem as a whole for your project if you don’t at least add an 
entry for it there. Given that we have this dynamic, we need to be particularly 
careful how the features we add can be used against people, particularly 
against the most vulnerable people in our community.

We sadly live in a world where our industry is incredibly toxic to, well 
basically everyone but white guys and are actively hostile towards efforts to 
seeing a community become more inclusive. These are people who will regularly 
create multiple twitter accounts in order to spam harassment at people (in 140 
characters) to get around cases where the person has blocked them. These are 
people who will flood comments on GitHub issue trackers for projects they don’t 
even use to bitch about someone changing some pronouns to be more inclusive.

Consider that a rude comment can completely crush someone’s motivation to learn 
Python, or to maintain a package. It can make our community seem all that more 
hostile and I don’t think the vast majority of comments are going to actually 
be very useful. I suspect they will largely be used as yet another support 
venue for random users who are confused (and deleting them doesn’t help those 
users either).

We had a comments and review system years ago (before my time TBH) and the 
backlash against it was so great that it was a major point of contention on 
catalog-sig where Package authors wanted it to be gotten rid of and the 
maintainers at the time pushing back to keep it. We (obviously) eventually got 
rid of it, and I think that is pretty indicative of the idea in general.

Any sort of user created content requires us, the people running PyPI, to 
moderate to some degree. We have to do it now with people who create projects 
with vulgar or offensive names and I don’t believe that we have the man power 
available to us to moderate the comments of a much larger feature that is going 
to incentivize people to make negative comments (and let’s be real, 95% of the 
comments are going to be negative, people rarely reach out to say they’re happy 
but they’re always ready to complain).

This is a lot of words to say that I would be very against this kind of feature 
on PyPI. I am not *entirely* against some sort of automated marker for possibly 
unmaintained packages, but even that I’m sketchy on. Allowing people to poop 
their own content onto project pages for a project they don’t own is just not 
tenable I think.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Ionel Cristian Mărieș via Distutils-SIG]

On Fri, Jul 15, 2016 at 1:51 AM, Steve Dower  wrote:

> This is why I listed a set of restrictions to help prevent that:
>
> * 140 chars (flexible, but short enough to prevent rants)
>

​Did you mean to write "provoke"​ instead of "prevent"? If we can learn one
thing from Twitter it's that such limit favors short and brutish comments
over the more nuanced and thoughtful ones - that take way more character
space of course.

I don't get what all this fuss is about about comments on PyPI. Such
feature seems unnecessary. There are plenty of ways to assess how well
maintained a package is. If a package maintainer wants comments or feedback
there's the url/long_description fields.


Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 14Jul2016 0619, Daniel D. Beck wrote:

Free-form, user-generated content on PyPI would become a pathway for
harassment and abuse. Introducing user-generated content on PyPI would
necessarily put an emotional burden on package maintainers in addition
to the maintenance burden (unless PyPI moderators are going to screen
content before maintainers and users see it—given the dearth of
resources for PyPI as it is, this strikes me as exceedingly unlikely).


This is why I listed a set of restrictions to help prevent that:

* 140 chars (flexible, but short enough to prevent rants)
* users must be logged in
* no external links
* maintainers can delete/dispute comments
* clear comments on each new release
* one comment per user per package (implied, but I didn't explicitly 
call it out in my previous email)


Do you really think this will be worse than the current state, where 
abusers *only* have access Twitter, github, reddit and email to harass 
package maintainers?


Assuming harassment is not going to be a problem, is there value in 
letting people add comments directly on the page where users seem to 
keep ending up?


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add data-requires-python attribute to download links in simple repository API

[Daniel Holth]

Lgtm

On Thu, Jul 14, 2016, 16:49 Thomas Kluyver  wrote:

> In a discussion about how to allow pip to select a package version
> compatible with the target Python version, Donald suggested adding a
> data-requires-python attribute to links in the simple repository API,
> which pip uses to find candidate downloads.
>
> ...
>
> This would expose the Requires-Python metadata field already specified
> in PEP 345. There is a separate PR for setuptools to allow specifying
> this value.
>
> I have opened a PR to add this to PEP 503, which defines the simple
> repository API, and Donald asked that I post about it here for comments:
>
> https://github.com/python/peps/pull/56
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Daniel D. Beck]

On Thu, Jul 14, 2016 at 11:57 AM, Wes Turner  wrote:

> Adding an embedded JS comments widget does/would add some additional
> maintenance burden (because user-generated content).


Free-form, user-generated content on PyPI would become a pathway for
harassment and abuse. Introducing user-generated content on PyPI would
necessarily put an emotional burden on package maintainers in addition to
the maintenance burden (unless PyPI moderators are going to screen content
before maintainers and users see it—given the dearth of resources for PyPI
as it is, this strikes me as exceedingly unlikely).


—Daniel
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Add data-requires-python attribute to download links in simple repository API

[Thomas Kluyver]

In a discussion about how to allow pip to select a package version
compatible with the target Python version, Donald suggested adding a
data-requires-python attribute to links in the simple repository API,
which pip uses to find candidate downloads.

...

This would expose the Requires-Python metadata field already specified
in PEP 345. There is a separate PR for setuptools to allow specifying
this value.

I have opened a PR to add this to PEP 503, which defines the simple
repository API, and Donald asked that I post about it here for comments:

https://github.com/python/peps/pull/56

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Jeremy Stanley]

On 2016-07-14 17:23:24 +0100 (+0100), Robin Becker wrote:
> not really sure why this is an issue. All of my problems are in virtual
> environments and I never use the --system-site-packages flag.
> 
> I have never used pip to install anything system wide (so far as I know).

Ahh, whoops, after rereading back through some of your earlier
messages in the thread I see this is indeed all in the scope of a
virtualenv.

> Step 0 in after setting up an environment is pip install -U pip setuptools
> 
> Of course you are right in that the python is a distro provided one; and
> also the pip and virtualenv etc etc.
[...]

I've noticed in the past that for some reason the system
pkg_resources can end up getting used by setuptools (on
Debian/Ubuntu it's unvendored and split into a separate
python-pkg-resources deb). I haven't had an opportunity to track it
down. You might try myenv/bin/python -c 'import sys;print sys.path'
and then check the results in order to see which the first one is to
provide pkg_resources.

Hunting around a bit, this looks similar to
https://github.com/pypa/setuptools/issues/497 so see if any of the
suggestions mentioned there help at all.

> When I run python -mvirtualenv I end up with an environment that has pip and
> setuptools already. Are you saying I should do
> 
> /usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv
> myenv/bin/python get_pip.py
[...]

No, but you might try bootstrapping a newer version of virtualenv
into its own virtualenv, like:

virtualenv foo
foo/bin/pip install -U pip setuptools virtualenv
foo/bin/virtualenv myenv

Doing that on some systems where I'm forced to start from
distro-provided pip/setuptools/virtualenv has worked out
consistently well.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Daniel Holth]

It's from Debian. They had time to break pip, but they don't have time to
fix it again.

On Thu, Jul 14, 2016 at 12:39 PM Paul Moore  wrote:

> On 14 July 2016 at 17:23, Robin Becker  wrote:
> > I used always to build python from source in older ubuntus, but that was
> > because we wanted the latest python 2.x etc etc. Using a local copy
> prevents
> > the os from smashing stuff, but means more work whenever a serious
> upgrade
> > is required.
> >
> > When I run python -mvirtualenv I end up with an environment that has pip
> and
> > setuptools already. Are you saying I should do
> >
> > /usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv
> > myenv/bin/python get_pip.py
> >
> > and then proceed from there? Or is it foolish to rely on the system
> python
> > at all?
> >
> > I haven't seen this problem in ubuntu 14.04, but that may be just luck.
> >
> > I certainly notice some new behaviour ie the system pip seems to want to
> > assert --user whereas it used to fail for lack of rights in installing
> into
> > /usr/lib/python
>
> Yes, apparently Ubuntu have patched pip to make --user the default.
> I've seen some bug reports caused by this patch, so it's possible that
> it's what is causing you problems here. Unfortunately, as that is an
> Ubuntu patch you'd need to report it to them.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Paul Moore]

On 14 July 2016 at 17:23, Robin Becker  wrote:
> I used always to build python from source in older ubuntus, but that was
> because we wanted the latest python 2.x etc etc. Using a local copy prevents
> the os from smashing stuff, but means more work whenever a serious upgrade
> is required.
>
> When I run python -mvirtualenv I end up with an environment that has pip and
> setuptools already. Are you saying I should do
>
> /usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv
> myenv/bin/python get_pip.py
>
> and then proceed from there? Or is it foolish to rely on the system python
> at all?
>
> I haven't seen this problem in ubuntu 14.04, but that may be just luck.
>
> I certainly notice some new behaviour ie the system pip seems to want to
> assert --user whereas it used to fail for lack of rights in installing into
> /usr/lib/python

Yes, apparently Ubuntu have patched pip to make --user the default.
I've seen some bug reports caused by this patch, so it's possible that
it's what is causing you problems here. Unfortunately, as that is an
Ubuntu patch you'd need to report it to them.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Robin Becker]

On 14/07/2016 16:14, Jeremy Stanley wrote:

On 2016-07-14 16:01:23 +0100 (+0100), Robin Becker wrote:

On 14/07/2016 15:41, Ian Cordasco wrote:

Try:

.


I would like to try and understand this happens as then I might have some
wya of fixing it.


You really should avoid mixing pip-installed packages in the system
context with distro-provided Python libraries, otherwise you will
run into these sorts of issues constantly. I help maintain some


not really sure why this is an issue. All of my problems are in virtual 
environments and I never use the --system-site-packages flag.


I have never used pip to install anything system wide (so far as I know).

Step 0 in after setting up an environment is pip install -U pip setuptools

Of course you are right in that the python is a distro provided one; and also 
the pip and virtualenv etc etc.




very, very large test infrastructure for Python-based tools and
libraries: in scenarios where we use pip to install anything
system-wide we first make sure to scrub every last distro-provided
Python library from the system along with any other Python-based
applications that might depend on them, and only then we bootstrap
pip completely independent of distro packaging (downloading and
running get-pip.py). Also whenever possible, we instead rely on pip
install within virtualenvs _without_ --system-site-packages, so that
there's no risk of interaction with any Python libraries that might
somehow get subsequently installed on the system.

I used always to build python from source in older ubuntus, but that was because 
we wanted the latest python 2.x etc etc. Using a local copy prevents the os from 
smashing stuff, but means more work whenever a serious upgrade is required.


When I run python -mvirtualenv I end up with an environment that has pip and 
setuptools already. Are you saying I should do


/usr/bin/python -mvirtualenv --no-pip --no-setuptools myenv
myenv/bin/python get_pip.py

and then proceed from there? Or is it foolish to rely on the system python at 
all?

I haven't seen this problem in ubuntu 14.04, but that may be just luck.

I certainly notice some new behaviour ie the system pip seems to want to assert 
--user whereas it used to fail for lack of rights in installing into 
/usr/lib/python

--
Robin Becker
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Jeremy Stanley]

On 2016-07-14 16:01:23 +0100 (+0100), Robin Becker wrote:
> On 14/07/2016 15:41, Ian Cordasco wrote:
> >Try:
> >
> >pip install --force-reinstall setuptools -U
> >
> 
> I didn't do the force-reinstall and for some reason when I cleaned both
> ~/.cache/pip and ~/.pip the pip install -r requirements.txt did work.
> 
> I have tried various solutions proposed in the past eg
> 
> sudo apt-get install --reinstall python-pkg-resources
> 
> but nothing seems to work.
> 
> I did the cache cleanups in desperation mode.
> 
> I did try pip install -U setuptools, but it says it is up to date.
> 
> If this might be a case of the underlying python changing on the server I
> have turned off automatic security updates.
> 
> I would like to try and understand this happens as then I might have some
> wya of fixing it.

You really should avoid mixing pip-installed packages in the system
context with distro-provided Python libraries, otherwise you will
run into these sorts of issues constantly. I help maintain some
very, very large test infrastructure for Python-based tools and
libraries: in scenarios where we use pip to install anything
system-wide we first make sure to scrub every last distro-provided
Python library from the system along with any other Python-based
applications that might depend on them, and only then we bootstrap
pip completely independent of distro packaging (downloading and
running get-pip.py). Also whenever possible, we instead rely on pip
install within virtualenvs _without_ --system-site-packages, so that
there's no risk of interaction with any Python libraries that might
somehow get subsequently installed on the system.
-- 
Jeremy Stanley
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Robin Becker]

On 14/07/2016 15:41, Ian Cordasco wrote:

Try:

pip install --force-reinstall setuptools -U



I didn't do the force-reinstall and for some reason when I cleaned both 
~/.cache/pip and ~/.pip the pip install -r requirements.txt did work.


I have tried various solutions proposed in the past eg

sudo apt-get install --reinstall python-pkg-resources

but nothing seems to work.

I did the cache cleanups in desperation mode.

I did try pip install -U setuptools, but it says it is up to date.

If this might be a case of the underlying python changing on the server I have 
turned off automatic security updates.


I would like to try and understand this happens as then I might have some wya of 
fixing it.



On Thu, Jul 14, 2016 at 8:37 AM, Robin Becker  wrote:

(myenv) rptlab@app0:~/myenv

.

Robin Becker

--
Robin Becker
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 14, 2016, at 5:30 AM, Михаил Голубев  wrote:
> 
> Ok, you convinced me that these extra requests from PyCharm won't cause you 
> any problems. Impressive stats, by the way :)
> 
> We will focus on migrating our packaging-related features to these new 
> endpoints; hopefully, it won't take long. Note, however, that we need to 
> prepare updates for already released versions of PyCharm. We'll let you know 
> as soon as everything is ready.
> 
> Ernest W. Durbin III suggested changing User-Agent, so that it's clear which 
> requests come from PyCharm. To me it seems a fair point.
> 
> Batch API, as mentioned by Steve Dower, are very welcome, anyway. Also 
> "/simple" index is still HTML page. Honestly, it's a bit cumbersome that this 
> information can be received only by scraping HTML and for everything else 
> there are JSON REST API and XML-RPC.

Yea, I plan on a new “next gen” API in Warehouse at some point that will be 
much cleaner overall and not require multiple different formats to use :). For 
the record, XML-RPC should be avoided where possible as well, we also can’t 
cache that in the CDN (because it’s a POST request to the same URL for all 
routes, and the CDN can’t inspect the body of a POST request to determine cache 
key). 

> 
> Is anyone from PyPA attending to EuroPython next week? We could discuss these 
> matters further there.

I’m not. I’m not sure if anyone else is.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] can someone explain why I have started to get these pip failures

[Ian Cordasco]

Try:

pip install --force-reinstall setuptools -U

On Thu, Jul 14, 2016 at 8:37 AM, Robin Becker  wrote:
>> (myenv) rptlab@app0:~/myenv
>> $ python
>> Python 2.7.11+ (default, Apr 17 2016, 14:00:29)
>> [GCC 5.3.1 20160413] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
>
>
>> (myenv) rptlab@app0:~/myenv
>>
>> $ pip --version
>> pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages
>> (python 2.7)
>
>
>
>> (myenv) rptlab@app0:~/myenv
>>
>>  pip install -r requirements.txt
>> Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1))
>>   Downloading MySQL-python-1.2.5.zip (108kB)
>> 100% || 112kB 2.1MB/s
>> Could not import setuptools which is required to install from a source
>> distribution.
>> Traceback (most recent call last):
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py",
>> line 372, in setup_py
>> import setuptools  # noqa
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py",
>> line 11, in 
>> from setuptools.extern.six.moves import filterfalse, map
>>   File "/home/rptlab/> (myenv)
>> rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py",
>> line 1, in 
>> from pkg_resources.extern import VendorImporter
>> ImportError: No module named pkg_resources.extern
>
>> (myenv) rptlab@app0:~/myenv
>>
>> $ pip install setuptools -U
>> Requirement already up-to-date: setuptools in
>> ./lib/python2.7/site-packages
>
>
>
> This is on a unbuntu 16.04lts system
>
>> $ uname -a
>> Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016
>> x86_64 x86_64 x86_64 GNU/Linux
>
>
> --
> Robin Becker
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] can someone explain why I have started to get these pip failures

[Robin Becker]

(myenv) rptlab@app0:~/myenv
$ python
Python 2.7.11+ (default, Apr 17 2016, 14:00:29)
[GCC 5.3.1 20160413] on linux2
Type "help", "copyright", "credits" or "license" for more information.




> (myenv) rptlab@app0:~/myenv

$ pip --version
pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages (python 2.7)



> (myenv) rptlab@app0:~/myenv

 pip install -r requirements.txt
Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1))
  Downloading MySQL-python-1.2.5.zip (108kB)
100% || 112kB 2.1MB/s
Could not import setuptools which is required to install from a source 
distribution.
Traceback (most recent call last):
  File "/home/rptlab/> (myenv) 
rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py", line 
372, in setup_py
import setuptools  # noqa
  File "/home/rptlab/> (myenv) 
rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py", line 11, in 

from setuptools.extern.six.moves import filterfalse, map
  File "/home/rptlab/> (myenv) 
rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py", line 1, 
in 
from pkg_resources.extern import VendorImporter
ImportError: No module named pkg_resources.extern

> (myenv) rptlab@app0:~/myenv

$ pip install setuptools -U
Requirement already up-to-date: setuptools in ./lib/python2.7/site-packages



This is on a unbuntu 16.04lts system


$ uname -a
Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 
x86_64 x86_64 GNU/Linux


--
Robin Becker
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecation of the endpoint "/pypi?%3Aaction=index"

[Nick Coghlan]

On 14 July 2016 at 05:05, Михаил Голубев  wrote:
> Hi guys. I'd like to clarify Dmitry's question a bit.
>
> We have some issues with suggested "/simple" endpoint. Despite the need to
> scrap the web page, old endpoint allowed us to quickly find latest versions
> of the packages hosted on PyPI. We did a single request on IDE startup and
> showed outdated installed packages in the settings later. Index "/simple"
> however contains only package names and links to the dedicated pages with
> their artifacts (not for each of them, though). It means that now we have to
> make tons of individual requests to find the latest published version for
> each installed package. Isn't it going to load the service even worse?

Donald addressed this in the other thread, but summarising it quickly
here (mainly for the benefit of the list archives):

- the "/simple" API is really cache friendly, and hence almost always
gets served out of the Fastly cache instead of hitting the main app
server
- the main index endpoint isn't cache friendly at all, so many (most?)
requests to it will hit the main app server
- that main app server is the weak link in the current setup that's
responsible for the 503 errors folks still sometimes see
- rendering the index page can be one of the culprits contributing to 503's

So if folks are using pypi.python.org to get this info, we'd prefer
multiple requests to the /simple API over a single request to the
index page.

However, Donald's also going to look at providing a suitable batch
query endpoint on https://pypi.io/ that folks can start using, even
before Warehouse officially launches (similar to the way we're already
recommending Warehouse as the preferred upload server).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Wes Turner]

There are types to describe this graph.

Thing > CreativeWork > SoftwareApplication

CreativeWork.comment r: [Comment]

http://schema.org/SoftwareApplication
http://schema.org/Comment

( #PEP426JSONLD because this is a graph of SoftwareApplication(s); now with
TOML metadata )

There could be edge types as well. e.g. what is the relation between
PIL/Pillow.
- maintainerSuggests
- communitySuggests
- communitySaysUnmaintained
- unaddressedVulns
- etc

Adding an embedded JS comments widget does/would add some additional
maintenance burden (because user-generated content).

Authors can specify an email address as structured data; and whatever they
consider relevant in the long_description.
On Jul 13, 2016 9:33 PM, "Steve Dower"  wrote:

On 13Jul2016 1456, Glyph Lefkowitz wrote:

>
> On Jul 13, 2016, at 1:54 PM, Steve Dower > > wrote:
>>
>> Possibly such user-contributed content would be valuable anyway
>>
>
> https://alternativeto.net but for PyPI? :)
>

Or just more general reviews/warnings/info. "Doesn't work with IronPython",
"Works fine on 3.5 even though it doesn't say so", etc.

Restrict it to 140 chars, signed in users, only allow linking to other PyPI
packages, let the maintainer delete comments (or mark them as disputed) and
I think you'd avoid abuse (or rants/detailed bug reports/etc.). Maybe
automatically clear all comments on each new release as well.

Doesn't have to be complicated and fancy - just enough that users can help
each other when maintainers disappear.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Михаил Голубев]

Ok, you convinced me that these extra requests from PyCharm won't cause you
any problems. Impressive stats, by the way :)

We will focus on migrating our packaging-related features to these new
endpoints; hopefully, it won't take long. Note, however, that we need to
prepare updates for already released versions of PyCharm. We'll let you
know as soon as everything is ready.

Ernest W. Durbin III suggested changing User-Agent, so that it's clear
which requests come from PyCharm. To me it seems a fair point.

Batch API, as mentioned by Steve Dower, are very welcome, anyway. Also
"/simple" index is still HTML page. Honestly, it's a bit cumbersome that
this information can be received only by scraping HTML and for everything
else there are JSON REST API and XML-RPC.

Is anyone from PyPA attending to EuroPython next week? We could discuss
these matters further there.







2016-07-13 23:54 GMT+03:00 Donald Stufft :

>
> On Jul 13, 2016, at 4:21 PM, Михаил Голубев  wrote:
>
> Can you handle that?
>
>
>
> Oh, and just to put things in scale in the past 30 days:
>
> * PyPI has served > 3 billion HTTP requests.
> * PyPI has served > 327TB of bandwidth.
> * The 95%tile for cache hit vs cache miss is 92%.
> * We regularly serve >1,000 concurrent requests -
> https://s.caremad.io/QDTlK0mRj7/
>
> —
> Donald Stufft
>
>
>
>


-- 
Best regards
Mikhail Golubev
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig