[Distutils] Specification for a local PyPI simple index
Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. And while I haven't really followed the offsite hosting changes, are the rel=homepage/rel=download links still as stated? (I think I'd want rel=download on everything as I only expect to provide URLs for actual package content). Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). I'm still thinking about design at the moment, so what I need is far from decided, but I want to be sure that I'm actually implementing the correct spec as a starting point! Thanks, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On Sep 9, 2013, at 10:54 AM, Paul Moore p.f.mo...@gmail.com wrote: Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? It might match what easy_install looks for but it's not up to date. I think it's still written as if the simple index had not been created. That being said it'll probably mostly work. One of my steps I want to do with warehouse is actually get this codified. For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. And while I haven't really followed the offsite hosting changes, are the rel=homepage/rel=download links still as stated? (I think I'd want rel=download on everything as I only expect to provide URLs for actual package content). MD5s are the only thing supported by PyPI but setuptools 0.9+ (I think?) and pip 1.2+ support sha1, sha256, sha512, etc. Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). This makes stuff like ``pip install django`` when the actual name is Django work. As long as your url matches safe_name it should be fine. I'm still thinking about design at the moment, so what I need is far from decided, but I want to be sure that I'm actually implementing the correct spec as a starting point! Thanks, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On 9 September 2013 16:11, Donald Stufft don...@stufft.io wrote: It might match what easy_install looks for but it's not up to date. I think it's still written as if the simple index had not been created. That being said it'll probably mostly work. One of my steps I want to do with warehouse is actually get this codified. Sigh. That's what I thought :-( Particularly as I don't actually want it to work with easy_install, but rather with pip. OK, I guess I'll just have to go with what that page and reverse-engineer from there. Warehouse (or at least something with an actual *spec*) can't come too soon for me... Thanks for the clarifications on the other points, too. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On Sep 9, 2013, at 11:15 AM, Paul Moore p.f.mo...@gmail.com wrote: On 9 September 2013 16:11, Donald Stufft don...@stufft.io wrote: It might match what easy_install looks for but it's not up to date. I think it's still written as if the simple index had not been created. That being said it'll probably mostly work. One of my steps I want to do with warehouse is actually get this codified. Sigh. That's what I thought :-( Particularly as I don't actually want it to work with easy_install, but rather with pip. OK, I guess I'll just have to go with what that page and reverse-engineer from there. Warehouse (or at least something with an actual *spec*) can't come too soon for me... Thanks for the clarifications on the other points, too. Paul It's not that complicated. Assuming an --index-url of https://pypi.python.org/simple/ ``pip install foo`` will look at 1. https://pypi.python.org/simple/foo/ 2. https://pypi.python.org/simple/ ``pip install foo==1.0`` will look at 1. https://pypi.python.org/simple/foo/1.0/ 2. https://pypi.python.org/simple/foo/ 3. https://pypi.python.org/simple/ It will stop once it finds a page that doesn't error or 404 (so typically it only looks at /foo/). Once it has a page it looks for html links and essentially categorizes them into installable and not installable. It's installable if it looks like a package url (the url looks like it's a file that matches the naming scheme). Then it looks at any rel = download or rel = homepage and if they are not installable links it will also fetch their HTML and do the same process of looking for installable links. Once it has a list of all the installable links for a page it selects the highest version one matching the spec we have and installs it. If there is a meta tag for api version and that version is = 2 then that triggers the new external stuff which is opt in for pip 1.4 and opt out in pip 1.5. This allows an index server to specify this file is hosted by me by adding a rel=internal to links to self hosted files. In pip 1.5 (and opt in for 1.4) pip will only install internal files (for servers with an api version metatag) unless the proper flags were passed to the command. There is a gotcha where setuptools (and pip still, though I want to deprecate it) also looks for a an extract string that looks like http://peak.telecommunity.com/DevCenter/EasyInstall#backward-compatibility which can be inside HTML comments. Does that help? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On 9 September 2013 16:27, Donald Stufft don...@stufft.io wrote: Does that help? It does. A lot. Many thanks for that, I could probably have worked most of it out but you've saved me a decent chunk of time. Cheers, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On Mon, Sep 9, 2013 at 10:54 AM, Paul Moore p.f.mo...@gmail.com wrote: Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? That spec is for setuptools 0.6. Later versions should have changed this documentation (in the PyPA repository) if they changed the protocol, but I don't know if anybody's actually keeping tabs on that. For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. Hm. Yeah, a quick glance at https://bitbucket.org/pypa/setuptools/src shows the docs unchanged, so whoever added the non-MD5 support forgot to check the docs for references to md5. And while I haven't really followed the offsite hosting changes, are the rel=homepage/rel=download links still as stated? (I think I'd want rel=download on everything as I only expect to provide URLs for actual package content). The meaning of re=downloadl values is, spider this page for download links, not this is a link to download.Links to download are identified by inspecting a link, not retrieving it. The only reason the rel tags exist is to mark URLs as spiderable. Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? It's definitive for easy_install. The only reason easy_install retrieves the root URL is if a requested package isn't found; the reason it does this is to catch alternative spellings due to case-sensitivity and/or differences in punctuation folding. If you can interpret easy_install's initial GET as a package requirement string (w/case- and punctuation-insensitivity via pkg_resources.safe_name()) rather than as an exact string match, then failure to match would produce the same failure to match on a full package listing, so there's no point having the full listing appear. It's strictly a fallback intended for dumb package indexes that simply consist of a directory tree and a web server providing directory listings. (I think it can even work with an FTP site, but it's been a while since I worked on that code.) The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). I don't know what pip does, but I assume that it's probably true of all package managers that either their targeted request succeeds or fails, and then they either request the full listing or they don't. So... the only possible way not providing the full list would be if some (foolish) package manager always began by requesting a full package listing. It's possible there are tools that wish to obtain a full listing and use the base URL for that... but AFAICT it would be a foolish thing to do if you're just trying to install packages. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
