On Jul 12, 2007, at 5:09 PM, Martin v. Löwis wrote: ... >> I wonder if it would help much to make multiple HTTP requests in the >> same connection. This might be something to look at in setuptools >> and/or xmlrpclib. > > Only for remote connections, due to the round-trips required for > TCP handshake. Locally, Apache opens a new connection to the FCGI > servers per requests (using the farmer-worker pattern).
Right, but most connections will be remote, so this is a potential win. > >> 1. Since many people will be far away from PyPI, I think our long- >> term plan should encompass geographic mirrors. It's good that the >> server is spending a small amount of time, but it still takes *me* a >> long time to get data. > > Ok. I am, in general, skeptical about mirroring. However, if it > makes people happy, feel free to implement it. My goal is to have PyPI provide a simplified version of the data for use by setuptools that is easily mirrored using standard mirroring tools. (I may actually prototype this with a kind of mirror.) > A number of issues should be considered, of course: > - there should be a way to get authoritative answers somehow, > preferably > from mirrors, but, if necessary, from the main site I don't know what you mean. I envision mirrors as being read-only and only used by setuptools. The main site would certainly be authoritative. > - I really wish to collect download counters across mirrors. > "Official" > mirrors should be obliged to report download statistics once a day > or so. OK. > >> 2. It's important to reduce the number of round trips. > > A colleague today suggested that the best way to reduce round trips > is to give each machine a local copy of the index, the same way > Debian apt works: you do 'apt-get update', and then have a local > copy of the catalog that you can build against. No roundtrips > at all (except for the one to update the local catalog), for the > expense of being out of date if you don't manually update the > catalog. Yup. This might be a really nice way to go. It would be especially nice if a client could contact PyPI and ask for new data since a given time. I imagine that this request could be as cheap as the requests we have now, unless a client was very out of date. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org _______________________________________________ Catalog-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/catalog-sig
