Re: [Distutils] environment markers in setup.cfg -- vs ;
On Wed, Sep 19, 2012 at 9:26 PM, Daniel Holth dho...@gmail.com wrote: I am implementing requirements-in-setup.cfg for wheel because there is no place to put environment markers in setup.py The Requires-Dist: lines in PKG-INFO use ; to separate environment markers from the dep name. Is setup.cfg supposed to use -- ? Why? requires-dist = keyring; python_version == '2.6' versus requires-dist = keyring -- python_version == '2.6' (Sorry, meant to reply-all; hate how that's not a default on python.org mailing lists.) Where does it say that? I see to recall it always using ; both in the PKG-INFO and in setup.cfg. I'm probably mistaken, but I'm just wondering where you got this from? And if so I wonder if it could be changed. The -- syntax is just confusing IMO. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] environment markers in setup.cfg -- vs ;
On Thu, Sep 20, 2012 at 1:32 PM, Erik Bray erik.m.b...@gmail.com wrote: On Wed, Sep 19, 2012 at 9:26 PM, Daniel Holth dho...@gmail.com wrote: I am implementing requirements-in-setup.cfg for wheel because there is no place to put environment markers in setup.py The Requires-Dist: lines in PKG-INFO use ; to separate environment markers from the dep name. Is setup.cfg supposed to use -- ? Why? requires-dist = keyring; python_version == '2.6' versus requires-dist = keyring -- python_version == '2.6' (Sorry, meant to reply-all; hate how that's not a default on python.org mailing lists.) Where does it say that? I see to recall it always using ; both in the PKG-INFO and in setup.cfg. I'm probably mistaken, but I'm just wondering where you got this from? And if so I wonder if it could be changed. The -- syntax is just confusing IMO. Erik Reading from http://www.python.org/~gbrandl/build/html2/packaging/setupcfg.html#extension-modules-sections Of course if you are always installing from wheel you would probably continue to specify anything build-related in Python, and only the install-time metadata from setup.cfg's [metadata] section would need environment markers. Wheel doesn't include setup.cfg or setup.py, just the .dist-info/METADATA generated from those files. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Sun, Sep 16, 2012 at 9:40 PM, Daniel Holth dho...@gmail.com wrote: I've been chatting to Chris McDonough a bit as well, and one potentially useful thing would be to clearly document the setuptools/distribute metadata precisely as it is generated today. The official docs for all egg formats can be found at: http://peak.telecommunity.com/DevCenter/EggFormats And they answer most of your questions. A few call-outs below: dependency_links.txt: url's of the package's dependencies. Speak up if you use this; it is very surprising, and has a much better replacement with pip --index options and requirements files. No, actually, those are not replacements at all, let alone better ones. They're for a package's supplier to offer third-party builds to fulfill requirements not available via PyPI. E.g. the author of package Foo offers unofficial Windows builds of package Bar, and includes dependency links so that those dependencies can be fulfilled even if not present on PyPI. Provides works the same way in setuptools, it is in PKG-INFO. Setuptools doesn't actually use Provides. Lexical formats of the files are described in terms of pkg_resources API calls -- these calls (and more precise syntax documentation) are documented here: http://peak.telecommunity.com/DevCenter/PkgResources Mainly in the Parsing Utilities section: http://peak.telecommunity.com/DevCenter/PkgResources#parsing-utilities ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
And they answer most of your questions. A few call-outs below: dependency_links.txt: url's of the package's dependencies. Speak up if you use this; it is very surprising, and has a much better replacement with pip --index options and requirements files. I will check to see how often this is used. No, actually, those are not replacements at all, let alone better ones. They're for a package's supplier to offer third-party builds to fulfill requirements not available via PyPI. E.g. the author of package Foo offers unofficial Windows builds of package Bar, and includes dependency links so that those dependencies can be fulfilled even if not present on PyPI. It's probably a style thing, but I like to distribute this information out-of-band. So as the author of foopackage (an application) I list all the requirements and their requirements and where they can be found, including git+http://github urls and so forth. I'm sure this has all been discussed to death over the last 6+ years. Provides works the same way in setuptools, it is in PKG-INFO. Setuptools doesn't actually use Provides. I noticed that. An installer could generate a file provided-name.egg-info based on this information. Ironically distribute would probably be the main client of this feature. Lexical formats of the files are described in terms of pkg_resources API calls -- these calls (and more precise syntax documentation) are documented here: http://peak.telecommunity.com/DevCenter/PkgResources Mainly in the Parsing Utilities section: http://peak.telecommunity.com/DevCenter/PkgResources#parsing-utilities I am impressed by pkg_resources. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Thursday, September 20, 2012 at 4:24 PM, Daniel Holth wrote: It's probably a style thing, but I like to distribute this information out-of-band. So as the author of foopackage (an application) I list all the requirements and their requirements and where they can be found, including git+http://github urls and so forth. I'm sure this has all been discussed to death over the last 6+ years. For what it's worth I agree with you here. I think that setup.(py|cfg) should point towards abstract requirements. (I need a distribution named foo) not concrete requirements (I need the distribution named foo from example.com). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Thu, Sep 20, 2012 at 4:53 PM, Donald Stufft donald.stu...@gmail.com wrote: On Thursday, September 20, 2012 at 4:24 PM, Daniel Holth wrote: It's probably a style thing, but I like to distribute this information out-of-band. So as the author of foopackage (an application) I list all the requirements and their requirements and where they can be found, including git+http://github urls and so forth. I'm sure this has all been discussed to death over the last 6+ years. For what it's worth I agree with you here. As a practical matter (and practicality beats purity), it's sometimes necessary to provide this information in order to depend on packages that aren't cataloged in PyPI, or when third-party build support is required, and your goal is to offer a library or application which itself is used as a dependency. That is, if project A depends on project B, which depends on C, which is not on PyPI, then A cannot simply depend on B without *copying* this out of band information, propagating a maintenance burden. Dependency links work around this problem (among others). So, IMO, this is actually a crucial part of making a composable dependency system in *practice*, even if it is not required in *theory*. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Thursday, September 20, 2012 at 9:24 PM, PJ Eby wrote: As a practical matter (and practicality beats purity), it's sometimes necessary to provide this information in order to depend on packages that aren't cataloged in PyPI, or when third-party build support is required, and your goal is to offer a library or application which itself is used as a dependency. This is likely the cause of the difference in opinion. I don't think that packaging metadata should contain information other than informative for dependencies external to the ecosystem. In this case If something isn't on PyPI then you don't get to have it automatically installed. If I depend on lxml (or somewhere in my dependency chain something does) then I have to convey that requirements for libxml2 and libxslt. I don't see things external to PyPI as any different. In metadata 1.2+ this could be expressed via requires-external. It's a dependency external to that of the Python packaging ecosystem. Maybe that's just me though.___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Upstream packaging feedback
This is a timely discussion for me because I need to get back to packaging some tools for RHEL/Centos that we have as dependencies. The tools are available on pypi or from a git repository. They seem to be available in older versions for Fedora 14 but not in EPEL or other sites for RHEL/Centos. One issue I encountered when previously working on the packaging for Fedora was that pypi packages do not necessarily follow the Linux Filesystem Hierarchy Standard. In fact, the packages may be implicitly organized for something like Windows where documentation goes in the same directory as code. I found I needed to use pypi2rpm bdist_rpm2 and modify the spec file to get things into the right places. BTW, the tools right now use python 2.7. So, please add the FHS issue to your list. I've also found that everything seems to be in a state of flux right now, with setuptools, distribute, distutils, distutils2, and pypi2rpm/bdist_rpm2 all out there and bumping into each other. When I tried to do a yum update to my Fedora 14 system, I ran into trouble because of conflicts among these packages. I hope this all settles down soon. Stan Klein On Thu, September 13, 2012 8:00 am, Toshio Kuratomi a.bad...@gmail.com wrote: Message: 1 Date: Wed, 12 Sep 2012 15:14:07 -0700 From: Toshio Kuratomi a.bad...@gmail.com To: Fedora Python SIG python-de...@lists.fedoraproject.org Subject: Re: Upstream packaging feedback On Wed, Sep 12, 2012 at 01:41:22PM +1000, Nick Coghlan wrote: I'll likely be helping to guide updates to the Python packaging format standards over the coming months. While they won't hit the standard library until 3.4, there will likely be third party tool support in earlier versions (since the whole point of the exercise is to eliminate the current implementation coupling to distutils and setuptools in favour of better defined metadata standards for communication between multiple tools). The first step will be reviewing the status quo and then creating a plausible road map (as well as describing current efforts for various aspects). I've started on that here: http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html One thing I would *love* to be able to enable is adding support for automatic mapping of PyPI distribution names (similar to what already exists for Perl and CPAN) where (for example), a developer could just write Requires: python(south) instead of having to figure out manually the name of the appropriate RPM package in Fedora. I believe that the new metadata fields defined in PEP 345 and PEP 426 should be enough to support that when generating a SPEC file from the PyPI metadata. Hey Nick! Thanks for working on this whole thing. I haven't been looking at things in the past year but was active on it a few years ago if you want to bounce any ideas around, get some idea of what's been discussed i nthe past by whom, or talk about what Fedora-specifically is currently doing/has found troublesome in the past. Mapping of pypi distribution names was something that I looked into a small bit with some Canonical people at PyCon two years ago. IIRC, it was part of trying to map distribution packages with each other and with pypi in order to figure out the state of python3 porting. Barry Warsaw still works there and might know more about what happened to it -- Allison Randal is who I was working with but I got the impression that she isn't working on that anymore so I don't know if she'll still have code around or not. There are numerous caveats to trying to do this, none of them insurmountable. I believe we were trying to compare versions as well as package names which was even tougher. Some things I can remember off the top of my head: * Multiple names for a package * pypi usually has the same name as the setup.py name field which is also encoded in the egg file name and metadata * The name of the module that is imported * One upstream package having multiple downstream names -- for instance, Debian has setuptools and pkg_resources as two separate binary packages. However, they're provided by the setuptools pypi entry. * A module that is provided by multiple upstreams: For instance, setuptools is provided by setuptools and distribute. pexpect is provided by pexpect and pexpect-u. * Some packages aren't present on pypi. Many library bindings provided by the libraries are this way, for instance libselinux-python. * Some libraries have conflicting names upstream (mock was one example in the past. ming still is one: http://www.libming.org/ and http://merciless.sourceforge.net/ * The naming for packages in the distribution isn't always simple. In Fedora, for instance, we have several styles and not all of them are mutually exclusive: * python-foo is the common case for libraries (python-docutils) * foo-python are the majority (but not all) of libraries which
Re: [Distutils] .egg-info metadata
On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com wrote: Maybe that's just me though. Nope, it's In the face of ambiguity, refuse the temptation to guess applied to dependency metadata. It's one of the core philosophical differences between the stdlib and systems like setuptools. If *others* want to guess in their automated systems, that's their prerogative. We prefer to put up a barrier and say be explicit about what you want. You see the same difference in perspective between datetime and dateutil - even though we agree there are contexts where guessing is useful, we will generally consider such tools a poor fit for the standard library, instead leaving them to the third party ecosystem. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Thu, Sep 20, 2012 at 10:51 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com wrote: Maybe that's just me though. Nope, it's In the face of ambiguity, refuse the temptation to guess applied to dependency metadata. How are dependency links guessing anything? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Fri, Sep 21, 2012 at 12:24 AM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Sep 21, 2012 at 1:58 PM, PJ Eby p...@telecommunity.com wrote: On Thu, Sep 20, 2012 at 10:51 PM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Sep 21, 2012 at 12:08 PM, Donald Stufft donald.stu...@gmail.com wrote: Maybe that's just me though. Nope, it's In the face of ambiguity, refuse the temptation to guess applied to dependency metadata. How are dependency links guessing anything? What are they a link to? What OSes will it run on? How do you install them? As you soon as you step outside the defined dependency framework, it's all guesswork. I'm not seeing how a documented file discovery protocol is guesswork. Perhaps you've not read the documentation? e.g.: http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall This is no more guesswork than the PyPI /simple index discovery protocol is. The only part I can think of in the whole process that is actually guesswork is interpreting distutils-generated (as opposed to setuptools-generated) distribution filenames -- a necessary nod to backward-compatibility. As far as the practicality vs. purity question, Python has already had Provides/Requires in the metadata format for several years, and it contained all the features that were needed for a pure dependency resolution system. In *theory* you could meet everyone's use cases with that, if you're going to assume that all packages live in PyPI and everyone has a compiler. And then there wouldn't be any need for any of the new metadata specs. In practice, however, not everything's in PyPI and not everybody has compilers, and not every package is instantly updated or maintained indefinitely. If you don't like how dependency links solve the problem, perhaps you could propose some other way of handling those use cases, such that developer B's dependency maintenance burdens are not propagated to developer A and his peer/parent dependers? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] .egg-info metadata
On Friday, September 21, 2012 at 12:57 AM, PJ Eby wrote: As far as the practicality vs. purity question, Python has already had Provides/Requires in the metadata format for several years, and it contained all the features that were needed for a pure dependency resolution system. In *theory* you could meet everyone's use cases with that, if you're going to assume that all packages live in PyPI and everyone has a compiler. And then there wouldn't be any need for any of the new metadata specs. That field only allows importable names which many distributions do not have (e.g. you can't put in `django-tastypie` into requires). The exact wording is: The format of a requirement string is identical to that of a module or package name usable with the 'import' statement, optionally followed by a version declaration within parentheses. These fields were _not_ for saying that it required a particular distribution/project and _were_ for saying it requires a particular module or package (in the import sense). In practice, however, not everything's in PyPI and not everybody has compilers, and not every package is instantly updated or maintained indefinitely. If you don't like how dependency links solve the problem, perhaps you could propose some other way of handling those use cases, such that developer B's dependency maintenance burdens are not propagated to developer A and his peer/parent dependers? I don't see why pushing the maintenance burden down the stack isn't an ok thing to do. We already do it for External requirements that either aren't python packages or are packaged in a way that standard tools cannot handle them. Obviously you have to draw the line somewhere between packages that you can depend on automatically and ones you can't. My problems with dependency_links is less to do with guessing and more about the fact it ties dependency installation to a particular host. The fact that dependencies are abstract requirements (a distribution named foo at version 2.0) and not a concrete location (distribution foo version 2.0 found at http://example.com/dists/) means that you get to choose at install time where your requirements come from. So a user could easily host their own intranet PyPI style server that they installed their dependencies from. A dependency that comes from dependency_links can't be installed from your own internal PyPI (or another public one) without rewriting the setup.py scripts (Unless my understanding of dependency links is wrong). Additionally by tying a dependency to an external system you decrease the overall installablity of a set of requirements* and you increase the number of locations that someone can use to exploit developers installing Python packages. These are important considerations when deciding to install or depend on a package and the dependency_links system hides it and makes it non obvious to end users what all is entailed in installing their set of requirements. (Note I have the same basic problem with external links on PyPI). I don't think it's entirely unreasonable for a packaging tool to have the ability to install from other locations that PyPI or a PyPI like server, I think that a packaging tool probably should have a dependency_links like system, however I think that the choice to use them should be in the hands of the person _installing_ the package, not the person creating the package. * If You depend on 5 different hosts for installing any particular set of requirements, and each of those 5 servers have a 99% uptime then the total combined uptime would be (0.99 * 0.99 * 0.99 * 0.99 * 0.99) = ~95%. In an average month you'd go from ~7 hours of uninstallable time per month (for one server at 99% uptime) to over a day of uninstallable time per month (for 5 servers at 99% uptime each). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
