Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 13Jul2016 1456, Glyph Lefkowitz wrote:



On Jul 13, 2016, at 1:54 PM, Steve Dower > wrote:

Possibly such user-contributed content would be valuable anyway


https://alternativeto.net but for PyPI? :)


Or just more general reviews/warnings/info. "Doesn't work with 
IronPython", "Works fine on 3.5 even though it doesn't say so", etc.


Restrict it to 140 chars, signed in users, only allow linking to other 
PyPI packages, let the maintainer delete comments (or mark them as 
disputed) and I think you'd avoid abuse (or rants/detailed bug 
reports/etc.). Maybe automatically clear all comments on each new 
release as well.


Doesn't have to be complicated and fancy - just enough that users can 
help each other when maintainers disappear.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Glyph Lefkowitz]

> On Jul 13, 2016, at 1:54 PM, Steve Dower  wrote:
> 
> Possibly such user-contributed content would be valuable anyway


https://alternativeto.net but for PyPI? :)

-glyph___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 13Jul2016 1252, Glyph Lefkowitz wrote:

The primary thing would be to have a banner on the page and a warning
from `pip install´.  Those of us close to the heart of the Python
community already have various ways of reading the tea leaves to know
that things are likely to be unmaintained or bitrotting; the main
purpose of such a feature would be to have an automated way for people
who /don't/ personally know all the prominent package authors and see
them at conferences and meetups all the time to get this information.
 For example: nobody should be using PIL, they should be using pillow.
 Yet there's no way for a new user to figure this out by just looking
at https://pypi.io/project/PIL/ :).

I think that the adjudication process for stealing a name from an
existing owner is something that still bears discussion, but separately.
 Whatever that process is, you'd have to go through it fully after a
package becomes thusly "abandoned", and for the reasons you cite, it
absolutely should not be automated.  Perhaps it shouldn't even be the
way to deal with it - maybe the most you should be able to do in this
case is to expand the "this is unmaintained" warning with a pointer to a
different replacement name.


I like this. Maybe if a maintainer doesn't trigger the switch/publish 
anything for a year, a banner appears on the page with a publicly 
editable (votable?) list of alternative packages - thinking something 
similar to a reviews system with an "I found this review helpful" button.


Possibly such user-contributed content would be valuable anyway, but the 
"probably abandoned" state just moves it to the top of the page instead 
of the bottom.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 13, 2016, at 4:21 PM, Михаил Голубев  wrote:
> 
> Can you handle that?


Oh, and just to put things in scale in the past 30 days:

* PyPI has served > 3 billion HTTP requests.
* PyPI has served > 327TB of bandwidth.
* The 95%tile for cache hit vs cache miss is 92%.
* We regularly serve >1,000 concurrent requests - 
https://s.caremad.io/QDTlK0mRj7/ 

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 13, 2016, at 4:21 PM, Михаил Голубев  wrote:
> 
> Right, sorry, that initial question wasn't clear about that. 
> 
> We need the latest versions only for installed packages. Nonetheless, as you 
> noted, it's still several dozens consecutive requests to 
> "/simple/" for each PyCharm session of every user. 
> 
> Can you handle that?


The short answer is yes.

The longer answer is, that we have Fastly acting as a CDN in front of PyPI and 
serving an item out of the cache in Fastly is essentially free for us in terms 
of resources (obviously Fastly needs to handle that load, but they’re well 
equipped to handle much larger loads than we are). Thus, the more cacheable 
(and the longer lived a particular cache item can be) the easier it is for us 
to scale a particular URL on PyPI.

The url you’re currently using has a view downsides that prevent it from being 
able to be cached effectively:

* The URL is a “UI” URL, so it includes information like current logged in user 
and thus we need to Vary: Cookie which means it’s less likely to be cached at 
all since each unique cookie header adds another response to be cached for that 
URL, and Fastly will only save ~200 responses per URL before it starts to evict 
some.

* Similarly to above, since it’s a “UI” URL people expect it to update fairly 
quickly, because legacy PyPI wasn’t implemented with long lived caching with 
purging on updates in mind, it was easier to simply implement it with a short 
(5 minute IIRC) TTL on the cached object rather than long lived TTLs with 
purging (as we do in the “API” urls).

* Responses that act as collections of projects need to be invalidated anytime 
something changes that may invalidate that collection. In an API that lists 
every project and the latest version, that means it needs to be invalidated 
anytime something releases a new version.

Compare that to looking at /simple/ and then either accessing /simple// or 
/pypi//json (all of which are cached for long periods of time and purged 
on demand).

* None of those are “UI” URLs, so they have long cache times and they do not 
Vary on Cookie.

* For /simple/ we don’t list any versions we only list projects themselves. 
This means that we only need to invalidate this page whenever a brand new 
project is added to PyPI or an existing project is completely deleted. This 
occurs far less than someone releasing an existing project.

* For /simple/ we don’t need to do any particularly heavy duty querying, it’s a 
simple select on an ~80k length table (versus a select on an 80k length table, 
with a join to a 500k length table) and is fairly quick to render.

* For /simple// and /pypi//json these are scoped to an individual 
project, so they can be cached for a very long time and only invalidated when 
that particular project releases, not when _any_ project releases. This means 
that the likelihood we can serve one of these out of cache is VERY high.

* For /simple// and /pypi//json our SQL queries are relatively quick 
because they don’t need to operate over the entire table, but only over the 
records for one single project.

Given all of the above, and the fact that listing every project and their 
latest version is *slow* and resource intensive, yes it’s very likely that 
doing that will be far better for our ability to serve your requests, because 
the extra requests will almost certainly be able to be served straight from the 
Fastly caches and never hit our origin servers at all.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Steve Dower]

I'm also interested (for the same support in Visual Studio) though we're 
unaffected by this change.

A batch API to get info for many packages would be great. Currently we scrape 
simple and then post JSON queries for individual packages.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Михаил Голубев" 
Sent: ‎7/‎13/‎2016 13:04
To: "Donald Stufft" 
Cc: "distutils-sig@python.org" 
Subject: Re: [Distutils] PyPI index workaround

I'm sorry, I should have posted my commentary here, not in the separate thread.
 
We have some issues with suggested "/simple" endpoint. Despite the need to 
scrap the web page, old endpoint allowed us to quickly find latest versions of 
the packages hosted on PyPI. We did a single request on IDE startup and showed 
outdated installed packages in the settings later. Index "/simple" however 
contains only package names and links to the dedicated pages with their 
artifacts (not for each of them, though). It means that now we have to make 
tons of individual requests to find the latest published version for each 
installed package. Isn't it going to load the service even worse?


So, yes, we're interested most in the latest version of a package. 


2016-07-13 21:57 GMT+03:00 Donald Stufft :



On Jul 13, 2016, at 2:43 PM, Dmitry Trofimov  
wrote:


Hi,


to have information about available packages, PyCharm IDE currently parses
the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index).
As it is going to be deprecated soon, we are looking for a workaround.


What we need is, making one request, to get the name and the version of all 
PyPI packages. Then we cache this information in the IDE 
(https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c84f88cc5edd5f9d1721/python/src/com/jetbrains/python/packaging/PyPIPackageUtil.java).


By name and version, do you mean the latest version?


—
Donald Stufft








___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig







-- 

Best regards
Mikhail Golubev___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Михаил Голубев]

Right, sorry, that initial question wasn't clear about that.

We need the latest versions only for installed packages. Nonetheless, as
you noted, it's still several dozens consecutive requests to
"/simple/" for each PyCharm session of every user.

Can you handle that?

2016-07-13 22:56 GMT+03:00 Donald Stufft :

>
> On Jul 13, 2016, at 3:40 PM, Dmitry Trofimov <
> dmitry.trofi...@jetbrains.com> wrote:
>
> Does that mean that PyPI index page will live for a while until the new
> API is implemented?
>
>
> Yes, though I’m looking at this right now.
>
> I do have a question here though. If I understand the dialog, this is to
> provide a way for people to upgrade packages they have installed, and to
> tell them if their is a newer version or not. So my question here is why do
> you need the latest version for *every* package instead of just the ones
> you have installed?
>
> If you narrow it down to just the ones that are installed, then the number
> of HTTP requests needed with the current APIs goes down from ~80,000 to
> likely <100 or even <50 in most cases.
>
> —
> Donald Stufft
>
>
>
>


-- 
Best regards
Mikhail Golubev
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Михаил Голубев]

I'm sorry, I should have posted my commentary here, not in the separate
thread.


> We have some issues with suggested "/simple" endpoint. Despite the need to
> scrap the web page, old endpoint allowed us to quickly find latest versions
> of the packages hosted on PyPI. We did a single request on IDE startup and
> showed outdated installed packages in the settings later. Index "/simple"
> however contains only package names and links to the dedicated pages with
> their artifacts (not for each of them, though). It means that now we have
> to make tons of individual requests to find the latest published version
> for each installed package. Isn't it going to load the service even worse?


So, yes, we're interested most in the latest version of a package.

2016-07-13 21:57 GMT+03:00 Donald Stufft :

>
> On Jul 13, 2016, at 2:43 PM, Dmitry Trofimov <
> dmitry.trofi...@jetbrains.com> wrote:
>
> Hi,
>
> to have information about available packages, PyCharm IDE currently parses
> the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index).
> As it is going to be deprecated soon, we are looking for a workaround.
>
> What we need is, making one request, to get the name and the version of
> all PyPI packages. Then we cache this information in the IDE (
> https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c84f88cc5edd5f9d1721/python/src/com/jetbrains/python/packaging/PyPIPackageUtil.java
> ).
>
>
> By name and version, do you mean the latest version?
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
Best regards
Mikhail Golubev
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Deprecation of the endpoint "/pypi?%3Aaction=index"

[Михаил Голубев]

Hi guys. I'd like to clarify Dmitry's question a bit.

We have some issues with suggested "/simple" endpoint. Despite the need to
scrap the web page, old endpoint allowed us to quickly find latest versions
of the packages hosted on PyPI. We did a single request on IDE startup and
showed outdated installed packages in the settings later. Index "/simple"
however contains only package names and links to the dedicated pages with
their artifacts (not for each of them, though). It means that now we have
to make tons of individual requests to find the latest published version
for each installed package. Isn't it going to load the service even worse?

-- 
Best regards
Mikhail Golubev
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 13, 2016, at 3:40 PM, Dmitry Trofimov  
> wrote:
> 
> Does that mean that PyPI index page will live for a while until the new API 
> is implemented? 

Yes, though I’m looking at this right now.

I do have a question here though. If I understand the dialog, this is to 
provide a way for people to upgrade packages they have installed, and to tell 
them if their is a newer version or not. So my question here is why do you need 
the latest version for *every* package instead of just the ones you have 
installed?

If you narrow it down to just the ones that are installed, then the number of 
HTTP requests needed with the current APIs goes down from ~80,000 to likely 
<100 or even <50 in most cases.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Glyph Lefkowitz]

> On Jul 12, 2016, at 9:54 PM, Donald Stufft  wrote:
> 
> 
>> On Jul 12, 2016, at 4:45 PM, Glyph Lefkowitz  wrote:
>> 
>> My feeling is that there should be a "dead man's switch" sort of mechanism 
>> for this.  Require manual intervention from at least one package owner at 
>> least once a year.  I believe if you dig around in the archives there's been 
>> quite a bit of discussion around messaging to package owners and that sort 
>> of thing - and the main sticking point is that someone needs to volunteer to 
>> do the work on Warehouse.  Are you that person? :)
> 
> 
> I suspect any change like this will require some sort of PEP or something 
> similar to it. It’s something that I think is going to hard to get just right 
> (if it’s something we want to do at all).
> 
> Software can be “finished” without needing more releases,

"The software isn't finished until the last user is dead." :-)

> and sometimes projects stop getting updates until the maintainer has more 
> time (or a new maintainer comes along).

Yes; the whole point here is to have some way for people to know that a new 
maintainer is needed.

> An example is setuptools which had no releases between Oct 2009 and Jun 2013.

Arguably setuptools _was_ badly broken though, and if it had been obvious 
earlier on that it was in a bad situation perhaps we'd be further along by now 
:-).

> Another nice example is ``wincertstore`` which has had two releases one in 
> 2013 and one in 2014 and is one of the most downloaded projects on PyPI. It 
> doesn’t need any updates because it’s just a wrapper around Windows APIs via 
> ctypes.

Except it does need testing against new versions of Python.  No Python :: 3.5 
classifier on it, for example!  And right at the top of its description, a 
security fix.

The point of such a switch is to be able to push it and respond; not to tell 
the maintainer "you have to do a new release!" but rather to prompt the 
maintainer to explicitly acknowledge "the reason I have not done a new release 
is not that I haven't been paying attention; I am alive, I'm paying attention, 
and we don't need any maintenance, someone is still watching".

> Another thing we need to be careful about is what do we do once said dead 
> man’s switch triggers? We can’t just release the package to allow anyone to 
> register it, that’s just pointing a security shaped footgun at the foot of 
> every person using that project? It doesn’t make sense to block new uploads 
> for that project since there’s no point to disallowing new uploads. Flagging 
> it to allow someone to “take over” (possibly with some sort of review) has 
> some of the security shaped footguns as well as a problem with deciding who 
> to trust with a name or not.

The primary thing would be to have a banner on the page and a warning from `pip 
install´.  Those of us close to the heart of the Python community already have 
various ways of reading the tea leaves to know that things are likely to be 
unmaintained or bitrotting; the main purpose of such a feature would be to have 
an automated way for people who don't personally know all the prominent package 
authors and see them at conferences and meetups all the time to get this 
information.  For example: nobody should be using PIL, they should be using 
pillow.  Yet there's no way for a new user to figure this out by just looking 
at https://pypi.io/project/PIL/ :).

I think that the adjudication process for stealing a name from an existing 
owner is something that still bears discussion, but separately.  Whatever that 
process is, you'd have to go through it fully after a package becomes thusly 
"abandoned", and for the reasons you cite, it absolutely should not be 
automated.  Perhaps it shouldn't even be the way to deal with it - maybe the 
most you should be able to do in this case is to expand the "this is 
unmaintained" warning with a pointer to a different replacement name.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Dmitry Trofimov]

>
>
> Ok, we don’t currently have an API like that (largely because nobody has
> come up with a use case that was pressing enough to need to devote
> resources to it). It was requested though, and is being tracked by
> https://github.com/pypa/warehouse/issues/347. This is likely enough to
> pull this issue onto my radar as sooner rather than later issue.


Does that mean that PyPI index page will live for a while until the new API
is implemented?

On Wed, Jul 13, 2016 at 9:25 PM, Donald Stufft  wrote:

>
> On Jul 13, 2016, at 3:12 PM, Михаил Голубев  wrote:
>
> I'm sorry, I should have posted my commentary here, not in the separate
> thread.
>
>
>> We have some issues with suggested "/simple" endpoint. Despite the need
>> to scrap the web page, old endpoint allowed us to quickly find latest
>> versions of the packages hosted on PyPI. We did a single request on IDE
>> startup and showed outdated installed packages in the settings later. Index
>> "/simple" however contains only package names and links to the dedicated
>> pages with their artifacts (not for each of them, though). It means that
>> now we have to make tons of individual requests to find the latest
>> published version for each installed package. Isn't it going to load the
>> service even worse?
>
>
> So, yes, we're interested most in the latest version of a package.
>
>
>
> Ok, we don’t currently have an API like that (largely because nobody has
> come up with a use case that was pressing enough to need to devote
> resources to it). It was requested though, and is being tracked by
> https://github.com/pypa/warehouse/issues/347. This is likely enough to
> pull this issue onto my radar as sooner rather than later issue.
>
>
> —
> Donald Stufft
>
>
>
>


-- 

Dmitry Trofimov
PyCharm Team Lead
JetBrainshttp://www.jetbrains.com
The Drive To Develop
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 13, 2016, at 3:12 PM, Михаил Голубев  wrote:
> 
> I'm sorry, I should have posted my commentary here, not in the separate 
> thread.
>  
> We have some issues with suggested "/simple" endpoint. Despite the need to 
> scrap the web page, old endpoint allowed us to quickly find latest versions 
> of the packages hosted on PyPI. We did a single request on IDE startup and 
> showed outdated installed packages in the settings later. Index "/simple" 
> however contains only package names and links to the dedicated pages with 
> their artifacts (not for each of them, though). It means that now we have to 
> make tons of individual requests to find the latest published version for 
> each installed package. Isn't it going to load the service even worse?
> 
> So, yes, we're interested most in the latest version of a package. 
> 


Ok, we don’t currently have an API like that (largely because nobody has come 
up with a use case that was pressing enough to need to devote resources to it). 
It was requested though, and is being tracked by 
https://github.com/pypa/warehouse/issues/347 
. This is likely enough to pull 
this issue onto my radar as sooner rather than later issue.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Donald Stufft]

> On Jul 13, 2016, at 2:08 PM, Dima Tisnek  wrote:
> 
> I'd rather see something similar to Linux distributions where there's
> a curated repository "core" and a few semi-official, like "extra" and
> "community," and for some, "testing."
> A name foobar resolves to core/foobar- if that exists, and if
> not some subset of other repositories is used.
> This way, an outdated package can be moved to another repo without
> breaking install base.


PyPI is unlikely to *ever* become a curated repository. The time and effort
it would take to do that, even if we decided we wanted to, is not something
that we have available to us.

Beyond that though, I think that would fundamentally change PyPI in a way
that is for the worse. One of the goals of PyPI is to enable anyone to publish
a package, whether they’re as well known and trusted as Guido, or some unknown
person from the backwoods of Pennsylvania. We try very hard to remain neutral
in terms of whether one package is “better” than another package and try to
present largely unbiased information [1].

It would not be particularly hard, technically speaking, for someone to
maintain a curated set of packages ontop of what PyPI provides already. This
would not need to be an official PyPI thing, but if one rose to some prominence
it would be easy enough to direct folks to it who want that sort of thing.


[1] To the extent that any information at all is unbiased.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Paul Moore]

On 13 July 2016 at 19:08, Dima Tisnek  wrote:
> I'd rather see something similar to Linux distributions where there's
> a curated repository "core" and a few semi-official, like "extra" and
> "community," and for some, "testing."
> A name foobar resolves to core/foobar- if that exists, and if
> not some subset of other repositories is used.
> This way, an outdated package can be moved to another repo without
> breaking install base.
>
> In fact, curation without namespaces will already be pretty good.

Who would take on the work of maintaining a curated repository? To do
anything like a reasonable job would be a *lot* of work.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Donald Stufft]

> On Jul 13, 2016, at 2:43 PM, Dmitry Trofimov  
> wrote:
> 
> Hi,
> 
> to have information about available packages, PyCharm IDE currently parses
> the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index 
> ).
> As it is going to be deprecated soon, we are looking for a workaround.
> 
> What we need is, making one request, to get the name and the version of all 
> PyPI packages. Then we cache this information in the IDE 
> (https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c84f88cc5edd5f9d1721/python/src/com/jetbrains/python/packaging/PyPIPackageUtil.java
>  
> ).

By name and version, do you mean the latest version?

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Dima Tisnek]

I came across a package by accident.
A mate made a reasonable mistake typing in a pip command, and
something odd got installed.
For a moment I even suspected that package in question was some kind
of malware, so I went to download it manually (not via pip install),
and realised that the package was not updated for a long while, didn't
have description and github link was broken.

That overall got me thinking about namespace pollution in pip, that
once something is pushed in, it's like to stay there forever. I
figured, with so many packages in pypi, what's the percentage that
cannot be removed by author for a simple reason that, in the worst
case, author is dead?

Btw., I'm not a fan of domain names (to finicky, change more often
that short names) or unique ids (humans don't handle them well).

I'd rather see something similar to Linux distributions where there's
a curated repository "core" and a few semi-official, like "extra" and
"community," and for some, "testing."
A name foobar resolves to core/foobar- if that exists, and if
not some subset of other repositories is used.
This way, an outdated package can be moved to another repo without
breaking install base.

In fact, curation without namespaces will already be pretty good.

d.

On 13 July 2016 at 19:24, Jim Fulton  wrote:
> On Tue, Jul 12, 2016 at 7:55 AM, Dima Tisnek  wrote:
>> Hi all,
>>
>> Is anyone working on pruning old packages from pypi?
>>
>> I found something last updated in 2014, which, looking at the source
>> appears half-done.
>> Github link doesn't work any longer, no description, etc.
>>
>> I managed to find author's email address out of band, and he responded
>> that he can't remember the password, yada yada.
>>
>> I wonder if some basic automation is possible here -- check if url's
>> are reachable and if existing package satisfies basic requirements,
>> failing that mark it as "possibly out of date"
>
> I'm curious why you view this as a problem that needs to be solved?
>
> - Do you want to take over the name yourself?
>
> - Are you afraid someone will stumble on this package and use it?
>
> Something else?
>
> Jim
>
> --
> Jim Fulton
> http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PyPI index workaround

[Dmitry Trofimov]

Hi,

to have information about available packages, PyCharm IDE currently parses
the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index).
As it is going to be deprecated soon, we are looking for a workaround.

What we need is, making one request, to get the name and the version of all
PyPI packages. Then we cache this information in the IDE (
https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c84f88cc5edd5f9d1721/python/src/com/jetbrains/python/packaging/PyPIPackageUtil.java
).

What official API could you advise us to look at?

Any hint is appreciated.


Best regards,
Dmitry
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Jim Fulton]

Well said.

IMO, package names shouldn't be reused.  Also, IMO, we have a
namespace problem, for which there's a common solution that we avoid
(domain based names, which can also be reused, but ...).

OTOH, here's an idea. What if in addition to the project name, we also
assigned a unique id. When a package was added to a consuming project,
we'd store both the packages name, and it's project id.  When looking
up a package, we'd supply both the project name and id.  If a name was
reused, the new project with the same name would have a new project id
and wouldn't be confused with the old one.  We could even still server
the old project's released without advertising them.  This way, if we
did decide to reuse a name, we could do so pretty safely.

Jim


On Wed, Jul 13, 2016 at 12:54 AM, Donald Stufft  wrote:
>
>> On Jul 12, 2016, at 4:45 PM, Glyph Lefkowitz  wrote:
>>
>> My feeling is that there should be a "dead man's switch" sort of mechanism 
>> for this.  Require manual intervention from at least one package owner at 
>> least once a year.  I believe if you dig around in the archives there's been 
>> quite a bit of discussion around messaging to package owners and that sort 
>> of thing - and the main sticking point is that someone needs to volunteer to 
>> do the work on Warehouse.  Are you that person? :)
>
>
> I suspect any change like this will require some sort of PEP or something 
> similar to it. It’s something that I think is going to hard to get just right 
> (if it’s something we want to do at all).
>
> Software can be “finished” without needing more releases, and sometimes 
> projects stop getting updates until the maintainer has more time (or a new 
> maintainer comes along). An example is setuptools which had no releases 
> between Oct 2009 and Jun 2013. Another nice example is ``wincertstore`` which 
> has had two releases one in 2013 and one in 2014 and is one of the most 
> downloaded projects on PyPI. It doesn’t need any updates because it’s just a 
> wrapper around Windows APIs via ctypes.
>
> Another thing we need to be careful about is what do we do once said dead 
> man’s switch triggers? We can’t just release the package to allow anyone to 
> register it, that’s just pointing a security shaped footgun at the foot of 
> every person using that project? It doesn’t make sense to block new uploads 
> for that project since there’s no point to disallowing new uploads. Flagging 
> it to allow someone to “take over” (possibly with some sort of review) has 
> some of the security shaped footguns as well as a problem with deciding who 
> to trust with a name or not.
>
> —
> Donald Stufft
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Missing IPv6 support on pypi.python.org

[Donald Stufft]

> On Jul 13, 2016, at 8:42 AM, Baptiste Jonglez  
> wrote:
> 
> As a follow-up, Fastly now provides an option to enable IPv6 (but this is
> not enabled by default).
> 
> See: https://github.com/pypa/pypi-legacy/issues/90#issuecomment-231240046
> 
> Does pypi plan to participate in this program?  It would be nice!


It’s not an option that we can turn on at will. It’s currently in Limited 
Availability and you have to ask Fastly to be put on a list and they’ll select 
which of their customers they want to invite into the LA program. We have asked 
to be put on the list and we’re awaiting to see if they accepted us into the LA.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Jim Fulton]

On Tue, Jul 12, 2016 at 7:55 AM, Dima Tisnek  wrote:
> Hi all,
>
> Is anyone working on pruning old packages from pypi?
>
> I found something last updated in 2014, which, looking at the source
> appears half-done.
> Github link doesn't work any longer, no description, etc.
>
> I managed to find author's email address out of band, and he responded
> that he can't remember the password, yada yada.
>
> I wonder if some basic automation is possible here -- check if url's
> are reachable and if existing package satisfies basic requirements,
> failing that mark it as "possibly out of date"

I'm curious why you view this as a problem that needs to be solved?

- Do you want to take over the name yourself?

- Are you afraid someone will stumble on this package and use it?

Something else?

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Brett Cannon]

On Tue, 12 Jul 2016 at 21:54 Donald Stufft  wrote:

>
> > On Jul 12, 2016, at 4:45 PM, Glyph Lefkowitz 
> wrote:
> >
> > My feeling is that there should be a "dead man's switch" sort of
> mechanism for this.  Require manual intervention from at least one package
> owner at least once a year.  I believe if you dig around in the archives
> there's been quite a bit of discussion around messaging to package owners
> and that sort of thing - and the main sticking point is that someone needs
> to volunteer to do the work on Warehouse.  Are you that person? :)
>
> [SNIP]
>
> Another thing we need to be careful about is what do we do once said dead
> man’s switch triggers? We can’t just release the package to allow anyone to
> register it, that’s just pointing a security shaped footgun at the foot of
> every person using that project? It doesn’t make sense to block new uploads
> for that project since there’s no point to disallowing new uploads.
> Flagging it to allow someone to “take over” (possibly with some sort of
> review) has some of the security shaped footguns as well as a problem with
> deciding who to trust with a name or not.


My assumption was that if a project was flagged as no longer maintained,
then it would literally just get some clear banner/label/whatever to let
people know that if they start using the project that they shouldn't
necessarily expect bug-fixes. And if people wanted to get really fancy,
expose this metadata such that some tool could easily warn you that you
have dependencies that have been flagged as unsupported code.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Missing IPv6 support on pypi.python.org

[Chris Withers]

  
  
Out of curiosity, how's that coming?


On 13/07/2016 14:08, Alex Grönholm
  wrote:


  
  The legacy software might have issues with IPv6 so I doubt this
  will happen before Warehouse replaces Cheeseshop as the new PyPI.
  
  13.07.2016, 15:42, Baptiste Jonglez
kirjoitti:
  
  
As a follow-up, Fastly now provides an option to enable IPv6 (but this is
not enabled by default).

See: https://github.com/pypa/pypi-legacy/issues/90#issuecomment-231240046

Does pypi plan to participate in this program?  It would be nice!

Thanks,
Baptiste

On Sun, Nov 08, 2015 at 09:13:49PM -0500, Donald Stufft wrote:


  I’m pretty sure that PyPI will get IPv6 support as soon as Fastly supports it and not any sooner. I know they’re working on making it happen but I don’t think they have a public timeline for it yet.

On November 8, 2015 at 4:34:32 PM, Baptiste Jonglez (bapti...@bitsofnetworks.org) wrote:

  
Hi,
 
pypi.python.org is currently not reachable over IPv6.
 
I know this issue was brought up before [1,2]. This is a real issue for
us, because our backend servers are IPv6-only (clients never need to talk
to backend servers, they go through IPv4-enabled HTTP frontends).
 
So, deploying packages from pypi on the IPv6-only servers is currently a
pain. What is the roadmap to add IPv6 support? It seems that Fastly has
already deployed IPv6 [3].
 
Thanks,
Baptiste
 
 
[1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html  
[2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity
[3] http://bgp.he.net/AS54113#_prefixes6
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
 

  
  -
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



  
  
  
  ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


  
  
  
  
  
  ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig



  

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Missing IPv6 support on pypi.python.org

[Alex Grönholm]

The legacy software might have issues with IPv6 so I doubt this will 
happen before Warehouse replaces Cheeseshop as the new PyPI.


13.07.2016, 15:42, Baptiste Jonglez kirjoitti:

As a follow-up, Fastly now provides an option to enable IPv6 (but this is
not enabled by default).

See: https://github.com/pypa/pypi-legacy/issues/90#issuecomment-231240046

Does pypi plan to participate in this program?  It would be nice!

Thanks,
Baptiste

On Sun, Nov 08, 2015 at 09:13:49PM -0500, Donald Stufft wrote:

I’m pretty sure that PyPI will get IPv6 support as soon as Fastly supports it 
and not any sooner. I know they’re working on making it happen but I don’t 
think they have a public timeline for it yet.

On November 8, 2015 at 4:34:32 PM, Baptiste Jonglez 
(bapti...@bitsofnetworks.org) wrote:

Hi,
  
pypi.python.org is currently not reachable over IPv6.
  
I know this issue was brought up before [1,2]. This is a real issue for

us, because our backend servers are IPv6-only (clients never need to talk
to backend servers, they go through IPv4-enabled HTTP frontends).
  
So, deploying packages from pypi on the IPv6-only servers is currently a

pain. What is the roadmap to add IPv6 support? It seems that Fastly has
already deployed IPv6 [3].
  
Thanks,

Baptiste
  
  
[1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html

[2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity
[3] http://bgp.he.net/AS54113#_prefixes6
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
  

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Missing IPv6 support on pypi.python.org

[Baptiste Jonglez]

As a follow-up, Fastly now provides an option to enable IPv6 (but this is
not enabled by default).

See: https://github.com/pypa/pypi-legacy/issues/90#issuecomment-231240046

Does pypi plan to participate in this program?  It would be nice!

Thanks,
Baptiste

On Sun, Nov 08, 2015 at 09:13:49PM -0500, Donald Stufft wrote:
> I’m pretty sure that PyPI will get IPv6 support as soon as Fastly supports it 
> and not any sooner. I know they’re working on making it happen but I don’t 
> think they have a public timeline for it yet.
> 
> On November 8, 2015 at 4:34:32 PM, Baptiste Jonglez 
> (bapti...@bitsofnetworks.org) wrote:
> > Hi,
> >  
> > pypi.python.org is currently not reachable over IPv6.
> >  
> > I know this issue was brought up before [1,2]. This is a real issue for
> > us, because our backend servers are IPv6-only (clients never need to talk
> > to backend servers, they go through IPv4-enabled HTTP frontends).
> >  
> > So, deploying packages from pypi on the IPv6-only servers is currently a
> > pain. What is the roadmap to add IPv6 support? It seems that Fastly has
> > already deployed IPv6 [3].
> >  
> > Thanks,
> > Baptiste
> >  
> >  
> > [1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html  
> > [2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity
> > [3] http://bgp.he.net/AS54113#_prefixes6
> > ___
> > Distutils-SIG maillist - Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >  
> 
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 


signature.asc
Description: PGP signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig