On May 9, 2014, at 4:12 AM, M.-A. Lemburg <m...@egenix.com> wrote: > On 08.05.2014 23:22, Donald Stufft wrote: >> >>> On a personal note, I'm uncomfortable with the way this change is >>> perceived as a case of *pip* enforcing a behaviour that the pip >>> developers feel should be required. I actually don't like this change >>> particularly. So having pip implement the behaviour required by that >>> PEP is to me simply a case of compliance with the agreed standard. But >>> now, as a pip developer, being held responsible for the resulting user >>> pain, and being expected to defend it, does not make me happy. >> >> I think the pain is being overrepresented and the positives are being >> ignored. >> The problem is the benefits of this PEP are much like the benefits of TLS >> too. For the vast majority of people they don’t notice anything different >> except >> installing things is faster and more reliable. They don’t attribute that to >> the >> PEP or this decision, they just internalize it as the new norm. However the >> people who this does affect will seek out why it broke and raise an issue >> citing that thing specifically. This creates a perception of lots of pain >> for no >> gain when the reality is not that. > > Donald: I don't think anyone is arguing that hosting packages on > PyPI is a bad thing and PyPI as a service has gotten a lot better > than it was a few years ago.
Didn’t mean to imply that I thought it was otherwise.
>
> However, I find it troubling that we as Python developers are *forcing*
> the whole Python world to put their code into PyPI.
Forcing is a strong word. As of right now we’re "forcing" you to put it onto
PyPI if you want to install it without *any* flags to pip. You're more then
capable of hosting it externally if you wish, and pip will even tell the people
who try to install it what they need to enable in order to install it. It even
allows people to say "I don't care about any of this crap, just make all the
external stuff work".
Even if pip removed the external link handling all together and required you
to do something like:
$ pip install --extra-index-url https://example.com/our-packages/ foo
$ pip install --find-links https://example.com/our-packages/ foo
We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
discouraging people from not hosting on PyPI and providing incentives to doing
that.
>
> There are plenty good reasons not to do this, and sometimes it's
> even impossible if you want to stay legal (see the PEP for details).
I re-read the reasons, I'm not sure I really agree with most (or all?) of them
however if people really want to do it, then there is nothing stopping them.
It's their choice and people on the *other* side of that who are installing
these packages also get to make a choice if they want to allow it or not.
>
> Accordingly, we should respect those reasons make it possible for
> Python packages to live elsewhere, without having our tools put
> those packages into a bad light or making it harder for Python
> users to install such packages than needed.
I'm not sure what "put it into a bad light" is supposed to mean here. Is it
about the warning? I've already removed that completely.
As far as making it harder than it "needs" to be, that's a hard line to draw. I
personally know people for whom things that are hosted externally have caused
them a lot of pain to the point where they have to automatically spider PyPI
for their own dependencies every hour or so in order to isolate themselves from
the failure of random dependencies suddenly no longer being installable.
This breakage is going to happen for basically any project that installs their
dependencies with any frequency. One of the big projects that I'm aware of that
has had this problem is Openstack who, in their CI, installs things several
hundred times a day. If a service they depend on goes down that causes
significant disruption for them. Now they've solved it by the above solution
of hosting their own mirror of PyPI that includes spidering for externally
installable things. They have one or two packages left that don't host directly
on PyPI at which point they'll no longer need that afaik.
I don't think telling every downstream of us of any size that in order to get
reliable installs they are going to have to work around PyPI, and in some cases
even develop custom software in order to do so effectively, is a very user
friendly position.
>
> With the checksum uploaded to PyPI, the only argument against
> fetching packages from other places on the Internet is reliability,
> not security.
I've never said differently. There are some minor privacy things in there
but primarily it's about reliability and that people should generally be aware
where their packages are coming from.
>
> PyPI is not the only reliable file hosting system on the Internet,
> so this argument is rather weak.
It actually doesn't matter how reliable the other systems are. Reliability wise
the best possible outcome is the external host has 100% (extremely unlikely)
uptime and it has no affect on reliability. The realistic outcome is that
the external hosts will not have 100% uptime and will infact decrease the
total availability of the install sets on PyPI.
This isn't an opinion it is fact.
But in reality reliability also includes more things than just the server
temporarily going down. It also includes things like the maintainer has quit
maintaining that package, left Python all together, died even. There are a
decent number of packages on PyPI that people use which have no active
maintainer at all. If people rely on a package that isn't hosted on PyPI they
take a larger risk that their dependencies are going to become uninstallable
due to the above. When scanning the unverifiable links it is amazing how many
of those links are now 404s or 500s or even "nodename not founds". All of these
cause issues for people and are not hypothetical, I've dealt with people
complaining about one or another of them.
When things are hosted on PyPI then we (we being the infra team) are able to
ensure that the likelihood of ``pip install <something>`` actually works and
continues to work. When things are hosted externally the best we can do is
say sorry. At that point the only major thing outside of our control that can
cause someones installations to suddenly start failing is if the package
author willfully removes their things from PyPI. That is unfortunate but it
is also their right and there's nothing to be done for that. Thankfully that
case is far less common than the case of "an external site just isn't there".
I think it's important to point out that one of the driving factors that caused
me to finally push for changes and what lead to PEP438 being created was that
Mercurial's external hosted was being extremely flaky. I can't remember the
exact details but I want to say that over the span of a week or two I was
getting massive numbers of users complaining that ``pip install Mercurial``
was suddenly failing. This isn't to knock on the Mercurial folks or anything
but to simply point out that these problems aren't things that just happen to
(under|un)maintained software nor are they hypothetical. This PEP was born of
the frustration that was being relayed to me by end users of PyPI/pip.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
