On 13 March 2013 07:18, Carl Meyer <c...@oddbird.net> wrote: > It seems to me that there's a remarkable level of consensus developing > here (though it may not look like it), and a small set of remaining open > questions. > > The consensus (as I see it):
I think that is a fair summary. One thing I'd like to mention, that I don't recall seeing so far is that PyPI is *really slow*. I don't mean 'the pypi web host is on a bad link' - far from it. pip, and I presume setuptools, spider to check dependencies and do the external HTML scraping and so forth. This takes an age when each new web host to talk to is a new DNS lookup (say 0.3 seconds) + HTTP request (0.6 seconds) with possible HTTPS setup in there too (up to 1.2 seconds). A project with dozens of dependencies in it's transitive dependency graph may take minutes *just spidering*. Now, if you read those figures and go 'zomg thats slow' - well yes, light speed isn't that fast - and even then while much of round-the-globe traffic is at light speed, a considerable chunk of time isn't. Moving all releases to one HTTPS host (and ensuring persistent connections are used for repeated index queries) [and then drop to HTTP for release files so they can be squid cached] is the simplest short term solution to this, and I'm *really* excited to see it being tackled. Longer term I'd love to see PyPI offer an API to return transitive data, to avoid the spidering altogether. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Cloud Services _______________________________________________ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
