On Jul 12, 2013, at 7:14 PM, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote:
> Donald Stufft <donald <at> stufft.io> writes: > >> I could probably be convinced about something that makes handling versions >> easier going into the standard lib, but that's about it. > > That seems completely arbitrary to me. Why just versions? Why not, for > example, support for the wheel format? Why not agreed metadata formats? As I said in my email, because it's more or less standalone and it has the greatest utility outside of installers/builders/archivers/indexes. > >> There's a few reasons that I don't want these things added to the stdlib >> themselves. >> >> One of the major ones is that of "agility". We've seen with distutils how >> impossible it can be to make improvements to the system. Now some of this > > You say that, but setuptools, the poster child of packaging, improved quite > a lot on distutils. I'm not convinced that it would have been as successful > if there were no distutils in the stdlib, but of course you may disagree. I've looked at many other languages where they had widely successful packaging tools that weren't added to the standard lib until they were ubiquitous and stable. Something the new tools for Python are not. So I don't think adding it to the standard library is required. And setuptools improved it *outside* of the standard library while distutils itself stagnated. I would venture to guess that if distutils *hadn't* been in the standard library than setuptools could have simply been patches to distutils instead of needing to be essentially "replace" distutils and it just so happened to reuse some it's functionality. So pointing towards setuptools just exposes the fact that improving it in the standard library was hard enough that it was done externally. > > I'm well aware of the "the stdlib is where software goes to die" school of > thought, and I have considerable sympathy for where it's coming from, but > let's not throw the baby out with the bathwater. The agility argument could > be made for lots of areas of functionality, to the point where you just > basically never add anything new to the stdlib because you're worried about > an inability to cope with change. Also, it doesn't seem right to point to > particular parts of the stdlib which were hard to adapt to changing > requirements and draw the conclusion that all software added to the stdlib > would be equally hard to adapt. Of course one could look at a specific piece > of software and assess its adaptability, but otherwise, isn't it verging on > just arm-waving? Well I am of the mind that the standard library is where software goes to die, and I'm also of the mind that a smaller standard library and a strong packaging story and ecosystem is far superior. But that's not what I'm advocating here. A key point to almost every other part of the standard library is if stagnates or falls behind or is unable to adapt then you simply don't use it. This is not a hard thing to do for something like httplib, urllib2, urlib, etc because it's what people have *done* in projects like requests. One persons choice to use url lib in his software has little to no bearing on someone else who might choose to use requests. However a packaging system needs interoperability. My choice to use a particular package software, if there is no interoperability, DRASTICALLY affects you if you want to use my software at all. A huge thing i've been trying to push for is decoupling packaging from a specific implementation so that we have a "protocol" (ala HTTP) and not a "tool" (ala distutils). However the allure of working to the implementation and not the standard is fairly high when there is a singular blessed implementation. > >> is made better with the way the new system is being designed with versioned >> metadata but it doesn't completely go away. We can look at Python's past to >> see just how long any individual version sticks around and we can assume that >> if something gets added now that particular version will be around for a long >> time. > > That doesn't mean that overall improvements can't take place in the stdlib. > For example, getopt -> optparse -> argparse. It's funny you picked and example where improvements *couldn't* take place and the entire system had to be thrown out and a new one written. getopt had to become a new module named opt parse, which had to become a new module named argparse in order to make changes to it. I don't think we need to have distutils, distlib, futurelib, even-further-futurelib and I think that makes packaging even more confusing then it needs to be. This also ties in with the above where one persons use of getopt instead of argparse doesn't drastically affect another person using a different one. > >> Another is because of how long it can take a new version of Python to become >> "standard", especially in the 3.x series since the entire 3.x series itself >> isn't standard, any changes made to the standard lib won't be usable for >> years and years. This can be mitigated by releasing a backport on PyPI, but >> if every version of Python but the latest one is going to require installing >> these libs from PyPI in order to usefully interact with the "world", then you >> might as well just require all versions of Python to install bits from PyPI. > > Well, other approaches have been looked at - for example, accepting things > into the stdlib but warning users about the provisional nature of some APIs. Provisional API's still exist in that version of Python and the only way someone would get a new one is by installing a package. I think that this makes the problem even *worse* because now you're adding API's to the standard library that have a good chance of needing to change and needing to require people to install a package (with no good way to communicate to someone that they need to update it since it's a standard library package and not a versioned installed package). > > I think that where interoperability between different packaging tools is > needed, that's where the argument for something in the stdlib is strongest, > as Brett said. You can gain interoperability in a few ways. One way is to just pick an implementation and make that the standard. Another is to define *actual* standards. The second one is harder, requires more thought and work. But it means that completely different software can work together. It means that something written in Ruby can easily work with a python package without shelling out to Python or without trying to copy all the implementation details and having to guess which ones are significant or not. > >> Yet another is by blessing a particular implementation, that implementations >> behaviors become the standard (indeed the way the PEP system generally works >> for this is once it's been added to the standard lib the PEP is a historical >> document and the documentation becomes the standard). However packaging is > > That's because the PEP is needed to advocate the inclusion in the stdlib and > as a record of the discussion and rationale for accepting/rejecting whatever > was advocated, but there's no real benefit in keeping the PEP updated as the > stdlib component gets refined from its real-world exposure through being in > the stdlib. And that's fine for a certain class of problems. It's not that useful for something where you want interoperability outside of that tool. How terrible would it be if HTTP was "well whatever Apache does, that's what HTTP is". > >> not like Enums or urllibs, or smtp. We are essentially defining a protocol, >> one that non Python tools will be expected to use (for Debian and RPMs for >> example). We are using these PEPs more like a RFC than a proposal to include >> something in the stdlib. > > But we can assume that there will either be N different implementations of > everything in the RFCs from the ground up, by N different tools, or ideally > one canonical implementation in the stdlib that the tool makers can use (but > are not forced to use if they don't want to). You might say that if there > were some kick-ass implementation of these RFCs on PyPI people would just > gravitate to it and the winner would be obvious, but I don't see things > working like that. In the web space, look at HTTP Request/Response objects > as an example: Pyramid, Werkzeug, Django all have their own, don't really > interoperate in practice (though it was a goal of WSGI), and there's very > little to choose between them technically. Just a fair amount of duplicated > effort on something so low-level, which would have been better spent on > truly differentiating features. A singular blessed tool in the standard library incentivizes the standard becoming and implementation detail. I *want* there to be multiple implementations written by different people working on different "slices" of the problem. That incentivizes doing the extra work on PEPs and other documents so that we maintain a highly documented standard. It's true that adding something to the standard library doesn't rule that out but it provides an incentive against properly doing standards because it's easier and simpler to just change it in the implementation. > >> There's also the case of usefulness. You mention some code that can parse the >> JSON metadata and validate it. Weel assumingly we'll have the metadata for >> 2.0 set down by the time 3.4 comes around. So sure 3.4 could have that, but >> then maybe we release metadata 2.1 and now 3.4 can only parse _some_ of the >> metadata. Maybe we release a metadata 3.0 and now it can't parse any >> metadata. But even if it can parse the metadata what does it do with it? The >> major places you'd be validating the metadata (other than merely consuming >> it) is either on the tools that create packages or in PyPI performing checks >> on a valid file upload. In the build tool case they are going to either need >> to write their own code for actually creating the package or, more likely, >> they'll reuse something like distlib. If those tools are already going to be >> using a distlib-like library then we might as just keep the validation code >> in there. > > Is that some blessed-by-being-in-the-stdlib kind of library that everyone > uses, or one of several balkanised versions a la HTTP Request / Response? If > it's not somehow blessed, why should a particular packaging project use it, > even if it's technically up to the job? It's not blessed and a particular packaging project should use it if it fits their needs and they want to use it. Or they shouldn't use it if they don't want. Standards exist for a reason. So you can have multiple implementations that all work together. > >> Now the version parsing stuff which I said I could be convinced is slightly >> different. It is really sort of it's own thing. It's not dependent on the >> other pieces of packaging to be useful, and it's not versioned. It's also the >> only bit that's really useful on it's own. People consuming the (future) PyPI >> API could use it to fully depict the actual metadata so it's kind of like >> JSON itself in that regard. > > That's only because some effort has gone into looking at version > comparisons, ordering, pre-/post-/dev-releases, etc. and considering the > requirements in some detail. It looks OK now, but so did PEP 386 to many > people who hadn't considered the ordering of dev versions of > pre-/post-releases. Who's to say that some other issue won't come up that we > haven't considered? It's not a reason for doing nothing. I didn't make any claims as to it's stability or the amount of testing that went into it. My ability to be convinced of that stems primarily from the fact that it's sort of a side piece of the whole packaging infrastructure and toolchain and it's also a piece that is most likely to be useful on it's own. > >> The installer side of things the purist side of me doesn't like adding it to >> the standard library for all the same reasons but the pragmatic side of me >> wants it there because it enables fetching the other bits that are needed for >> "pip install X" to be a reasonable official response to these kind of >> questions. But I pushed for and still believe that if a prerequisite for >> doing that involves "locking" in pip or any of it's dependencies by adding >> them to the standard library then I am vehemently against doing it. > > Nobody seems to be suggesting doing that, though. I was (trying?) to explain that my belief doesn't extend to only distlib here and instead to the entire toolchain. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
