On Tuesday, August 23, 2016, Donald Stufft <don...@stufft.io <javascript:_e(%7B%7D,'cvml','don...@stufft.io');>> wrote:
> > > It’s hard to get per project numbers for these (or at least, it takes a > more > complex query than I can manage with my head here). However, I think it’s > pretty telling that when you start looking at other formats, not only is > the > primary consumer tools that just indiscriminately download everything from > PyPI, > but almost *all* of the consumers of those files are tools that just > indiscriminately download everything. Unless there are users of those > mirrors who > follow vastly different usage patterns than what we see on PyPI itself, > the primary > purpose of bdist_wininst, bdist_msi, bdist_dmg, etc on PyPI is to consume > disk space > and bandwidth via the mirroring infrastructure. Other ways to reduce mirroring bandwidth: Caching proxies: - devpi (proxy cache only what is needed) http://doc.devpi.net/latest/quickstart-pypimirror.html - "pypi proxy" https://www.google.com/search?q=pypiproxy - nginx caching proxy - https://gist.github.com/dctrwatson/5785638 - this would need to be updated for the pypi.org warehouse routes - it may be worth linking to these in the/some mirroring infrastructure docs Pip --offline switch - Internet was down the other day and i knew the packages were locally cached but I couldn't figure out how to make pip work w/ local offline dependency resolution More efficient mirroring implementations: - rsync - bup - DRPM ... delta downloads would be more efficient here: https://github.com/pypa/warehouse/issues/347 "Add API endpoint to get latest version of all projects" > I’d also like to note, that the numbers above are conservative on what they > consider to be a “mirroring client”. For instance, devpi used to use the > default > requests user-agent, and we see downloads via the requests user agent, but > did not > count them as mirroring clients because it could be some other script > doing the > downloading. > > — > Donald Stufft > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig