On Mon, Mar 4, 2013 at 1:54 AM, Mark McLoughlin <mar...@redhat.com> wrote: > I've tried to digest PEP426 and collected my thoughts below. Apologies > if you read through them and they offer you nothing useful. > > I'm trying to imagine the future we're driving towards here where > OpenStack is no longer suffering and how PEP426 helped get us there > > e.g. where > > - Many libraries use semantic versioning and OpenStack specifies x.y > compatible versions in their dependency lists. Incompatible changes > only get made in x+N.y and OpenStack continues using the x.y+N.z > versions.
Yep, PEP 426 pushes projects heavily in that direction by making it the path of least resistance. You *can* do something different if necessary, it's just less convenient than simply using the first 9 clauses of semantic versioning. > - All the libraries which don't use semantic versioning use another > clear versioning scheme that allows OpenStack to avoid incompatible > updates while still using compatible updates. Yes, the PEP will explicitly encourage projects to document how to define dependencies if the default "compatible release" clause isn't appropriate. > - Incompatible versions of the same library are routinely installed > in parallel. Does PEP426 here, or is all the work to be done in > tools like PyPI, pip, setuptools, etc. Apps somehow specify which > version they want to use since the incompatible versions of the > library use the same namespace. PEP 426 doesn't help here, and isn't intended to. virtualenv (and the integrated venv in 3.3+) are the main solution being offered in this space (the primary difference with simple bundling is that you still keep track of your dependencies, so you have some hope of properly rolling out security fixes). Fedora (at least) is experimenting with similar capabilities through software collections. IMO, the success of iOS, Android (and Windows) as deployment targets means the ISVs have spoken: bundling is the preferred option once you get above the core OS level. That means it's on us to support bundling in a way that doesn't make rolling out security updates a nightmare for system administrators, rather than trying to tell ISVs that bundling isn't supported. > There are fairly large gaps in my understanding of the plan here. How > quickly will we see libraries adopt PEP426? Will python 2.6/2.7 based > projects be able to consume these? PEP 426 is aimed primarily at the current ecosystem of tools (setuptools/distribute/pip/zc.buildout and others). There are limitations to the current PyPI API that prevent them from taking full advantage of the extended metadata, however, so full exploitation won't be possible until the server side gets sorted out (which is an ongoing discussion). > How do parallel installs work in > practice? etc. At the moment, they really don't. setuptools/distribute do allow it to some degree, but it can go wrong if you're not sufficiently careful with it (e.g. http://git.beaker-project.org/cgit/beaker/commit/?h=develop&id=d4077a118627b947a3c814cd3ff9280afeeecd73). > > Thanks, > Mark. > > == Basic Principles == > > What we (OpenStack) need from you (python library maintainers): > > 1) API stability or, at least, predictable instability. We'd like to > say "we require version N or any compatible later version" > because just saying "we require version N" means our users can't > use our application with later incompatible versions of your > library. I assume you mean s/incompatible/compatible/ in the last example. I agree, which is why PEP 426 is deliberately more opinionated than PEP 345 about what versions should mean (although I need to revise how that advocacy is currently handled). > 2) Support for installing multiple incompatible versions at > once[*]. If there are two applications in the same > system/distribution that require incompatible versions of the > same library, you need to support having both installed at > once. virtualenv/venv cover this (see http://www.virtualenv.org/en/1.9.X/ and http://www.python.org/dev/peps/pep-0405/ > == Versioning == > > Semantic versioning is appealing here because, assuming all libraries > adopt it, it becomes very easy for us to predict which versions will > be incompatible. > > For any API unstable library (0.x in semantic versioning), we need to > pin to a very specific version and require distributions to package > that exact version in order to run OpenStack. When moving to a newer > version of the library, we need to move all OpenStack projects at > once. Ideally, we'd just avoid such libraries. > > Implied in semantic versioning, though, is that it's possible for > distributions to include both version X.y.z and X+N.y.z My point of view is that the system Python is there primarily to run system utilities and user scripts, rather than arbitrary Python applications. Users can install alternate versions of software into their user site directories, or into virtual environments. Projects are, of course, also free to include part of their version number in the project name. The challenge of dynamic linking different on-disk versions of a module into a process is that: - the import system simply isn't set up to work that way (setuptools/distribute try to fake it by adjusting sys.path, but that can go quite wrong at times) - it's confusing for users, since it isn't always clear which version they're going to see - errors can appear arbitrarily late, since module loading is truly dynamic > == PEP426 == > > === Date Based Versioning === > > OpenStack uses date based versioning for its server components and > we've begun using it for the Oslo libraries described above. PEP426 > says: > > Date based release numbers are explicitly excluded from > compatibility with this scheme, as they hinder automatic translation > to other versioning schemes, as well as preventing the adoption > of semantic versioning without changing the name of the > project. Accordingly, a leading release component greater than or > equal to 1980 is an error. > > This seems odd and the rationale seems week. It looks like an > abritrary half-way house between requiring semantic versioning and > allowing any scheme of increasing versions. The assumption is that the version field will follow semantic versioning (that's not where the PEP started, it was originally non-committal on the matter like PEP 345 - however I've come to the conclusion that this assumption needs to be made explicit, but still need to update various parts of the PEP that are still wishy-washy about it). However, we can't actually enforce semantic versioning because we can't automatically detect if a project is adhering to those semantics. What we *can* enforce is the rejection of version numbers that obviously look like years rather than a semantic version. > I'm really not sure how we'd deal with this gracefully - assuming we > have releases out there which require e.g. oslo-config>=2013.1 and we > switch to: > > Version: 1.2 > Private-Version: 2013.2 > > then is the >=2013.1 requirement satisfied by the private version? If > not, should we just go ahead and take the pain now of releasing 2013.1 > as 1.1? No version of the metadata will ever support ordered comparisons of the "Private-Version" field, since that's an arbitrary tag with no implied ordering mechanism. I expect we'll add support for == and != comparisons against private tags somewhere along the line once PyPI is publishing the enhanced metadata properly. > === Compatible Release Version Specifier === > > I know this is copied from Ruby's spermy operator, but it seems to be > to be fairly useless (overly pessimistic, at least) with semantic > versioning. > > i.e. assuming x.y.z releases, where an increase in X means that > incompatible changes have been made then: > > ~>1.2.3 > > unnecessarily limits you to the 1.2.y series of releases. Now, in an > ideal world, you would just say: > > ~>1.2 > > and you're good for the entire 1.x.y but often you know what your app > doesn't work with 1.2.2 and there's a fix in 1.2.3 that your app > absolutely needs. Handling that kind of situation is why version specifiers allow multiple clauses that are ANDed together: mydependency (1.2, >= 1.2.3) > > A random implementation detail I don't get here ... we use 'pip install > -r' a lot in OpenStack and so have files with e.g. > > PasteDeploy>=1.5.0 > > how will you specify compatible releases in these files? What's the > equivalent of > > PasteDeploy~>1.5.0 > > ? Note that the pkg_resources specifier format (no parens allowed) is NOT the same as the PEP 345/426 format (parens required). The former is designed to work with arbitrary version schemes, while the latter is explicitly constrained to version schemes that abide by the constraints described in PEP 386. I expect the setuptools/distribute developers will choose a tilde-based operator to represent compatible release clauses in the legacy formats (likely Ruby's ~>, although it could be something like ~= or ~~ instead). > === Multiple Release Streams === > > If version 1.1.0 and version 1.2.0 were already published to PyPI, > we'd like to be able to release a 1.1.1 bugfix update. I saw some > mention that PyPI would then treat 1.1.1 as the latest release, but I > can't find where I read that now. > > Could someone confirm this won't be an issue in future. I guess I'm > unclear how the PEP relatest to problems with PyPI specifically. There are major issues with the way PyPI publishes metadata - in particular, installers must currently download and *run* setup.py in order to find out the dependencies for most packages. Dev releases are also prone to being picked up as the latest release, because there's no explicit guideline for how installers should identify and exclude development releases when satisfying dependencies. > === Python 2 vs 3 === > > Apparently, part of our woes are down to library maintainers releasing > a version of their library that isn't supported on Python 2 (OpenStack > currently supports running on 2.6 and 2.7). > > This is obviously an incompatible change from our perspective. Going > back the basic principles above, we'd like to be able to: > > 1) specify that we "require version N or any later compatible > version that supports python 2.6. and 2.7". You'll need to figure out which version broke compatibility and add a "< first-broken-version" clause to the version specifier for that dependency (pointing out to the offending upstream that it isn't cool to break backwards compatibility like that in a nominally backwards compatible release may also be worthwhile). However, once Requires-Python metadata is more readily available, installers should also automatically exclude any candidate versions with an unsatisfied "Requires-Python" constraint. Single source and 2to3 based Python 2.6+/3.2+ compatibility is unfortunately currently somewhat clumsy to declare, since version specifiers currently have no "or" notation: Requires-Python: >= 2.6, < 4.0, !=3.0.*, !=3.1.* Probably something to look at for metadata 2.1 (I won't delay metadata 2.0 over it, since single source compatibility *can* be specified accurately once I update the PEP to include the ".*" notation, it's just ugly). > 2) have both the python 2 and python 3 compatible versions installed > on the same system or included in the same distribution. > > Environment markers in PEP426 sound tantalizingly close to (1) but > AFAICT the semantics of markers are "the set of things that must be > true for this field to be considered" i.e. > > Requires-Dist: PasteDeploy (>=1.5.0); python_version == '2.6' or > python_version == '2.7' > > means PasteDeploy is required on python 2.6/2.7 installs, but not > otherwise. Correct. > I really don't know what the story with (2) is. virtualenv/venv/software collections. > > == Oslo Libraries == > > The approach we had planned to take to versioning for the Oslo family > of libraries in OpenStack is: > > - APIs can be marked as deprecated in a release, kept around for > another release and then removed. For example, if we deprecated > an API in the H release it would not be remove until 1 year later > in the J release. That actually matches the traditional deprecation model for CPython. However, it's hard to handle cleanly with automated dependency analysis because it's hard to answer the "is this a backwards compatible release?" question quickly and automatically, since the answer is "it should be, but may not be if the application is currently triggering deprecation warnings". Depending on how conservative people are, they'll either choose the more restrictive option of disallowing forward compatibility, or the optimistic approach of declaring forward compatibility with the rest of the 3.x series. > - If we wanted to go down the route of a version with incompatible > API changes with no deprecation period for the old APIs, we'd > actually just introduce the new APIs under a new library name > like oslo-config2. We could continue to support the old APIs as a > separate project for a time. > > This is an approach tailored to Oslo's use case as a library used by > server projects on a 6 month coordinated release cycle. It allows > servers from the H release to use Oslo libraries from the I release, > further allowing servers from the H and I release to be in the same > system/distribution. > > To encode this in a requirements file, I guess we'd do: > > oslo-config>=2013.1,<2014.1 > > although that does mean predicting that 2014.1 changes which are > incompatible with 2013.1, even though it may not actually do so. PEP 426 explicitly forbids you from doing that, since it disallows version numbers that look like dates. But you can add a leading zero to make it legal (I'll be changing the advice in the PEP to suggest this solution for date-based version compatibility): oslo-config (>= 0.2013.1, < 0.2014.1) I'll also be updating the "compatible release" clause specification to pay attention to leading zeroes so that "0.N" and "0.N+1" are considered incompatible by default (which again works in well with this approach). > This approach doesn't work so well for "normal" Python libraries > because it's impossible to predict when it will be reasonable to > remove a deprecated API. Who's to say that no-one will have a 3 year > old application installed on their system that they expect to still > run fine even if they update to a newer version of your library? Yes, we've become a lot more conservative with this even in the standard library. Semantic versioning is a better approach in general. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
