On October 5, 2015 at 2:29:35 AM, Nathaniel Smith ([email protected]) wrote: > > Does that sound right?
Off the bat, I'm perfectly fine with `pip install archive-made-from-vcs.tar.gz` working the same as `pip install git+https://github.com/foo/bar.git``. The fact that one is a VCS and one isn't is immaterial to me that's just a transport mechanism for an arbitrary collection of files and directories. These arbitrary collections of file and directories can only provide a very minimal amount of information about the thing that is contained inside of them, and indeed there may even be multiple things inside of this rooted within different sub directories of the top level arbitrary collection of files. I agree with Paul though, I don't see any way this arbitrary collection of files/directories can be distributed on PyPI. PyPI is not just a place to dump whatever random archives you want for a package, it's specifically a place for Python package files (sdists, wheels, and eggs right now for the most part). It sounds like maybe you're just looking for a way to make it so that pip no longer makes these "arbitrary files/directories" installs work by treating them like unpacked sdists and instead follows some supported path. If so, that is reasonable to me (and something I planned on getting around to). If that is what you're shooting for, I think it got confused by trying to mix in the sdist concept, as our sdists and wheels are not really for human consumption any more. I don't think that it makes sense for pip to go directly from a VCS[1] to a Wheel in the grand scheme of things. Right now we kind of do it, but that's because we just treat them like an unpacked sdist [2], long term though I don't think that is the correct way to do things. We (I?) want to minimize the different installation "paths" that can be taken and the main variable then when you do a ``pip install`` is how far long that path we already are. My ideal path looks something like this: VCS -> Source Wheel [3] -> Wheel -> Installed \-> Inplace Installation [4] So in that regards, I think that the only things (in my ideal world) people should be uploading to PyPI are source wheels (mandatory? [5]) and binary wheels (optional?). I don't know if downstream would prefer to use a source wheel or an arbitrary collection of files that may or may not be in a tarball, but I'm focused primarily on the needs of our toolchain, while still trying to make sure we don't end up in a situation that hurts downstream as well. I also don't think it makes sense for pip to ever really install these items, no matter what the transport mechanism is (VCS, tarball, unpacked on disk) without being explicitly pointed to by an URL or file path. There's obviously some backwards compatability issues here, because we can't just stop fetching .tar.gz links or anything, but I think the expectation should be that these items are only ever directly installed, not installed as part of the dependency resolution process. In that vein, they don't really participate in the dependency resolution process either (we'll install their dependencies and what not, but we'll assume that since you're pointing us to an explicit archive that we aren't going to resolve that particular dependency to anything other than what you've explicitly given us). If we're only installing these items when explicitly being given them by a direct URL or path, then a lot of the worries about non static metadata no longer exist, because as a user you're explicitly opting into installing something that isn't a real package, but is a VCS install of something that could be a package. Which we'll fetch, turn into a source wheel, and then turn that into a binary wheel (or do an inplace install). I also don't think it makes sense for these VCS installs to directly support outputting multiple different source wheels, but instead rely on the fact pip lets you say "install this artifact, but first CD into a particular directory". So you'd essentially just structure the filesystem of your VCS install to make independent little mini projects that could be independently packaged into their own VCS installs, but are all contained within some larger thing and we need to install a specific sub directory. I'm happy to be convinced otherwise, but given that this is a sort of edge case that a project will need this and we already have the subdirectory support I think it's simpler to just leverage that. Given my desire to reduce the number of installation paths we actually support, I think that trying to standardize this concept of a VCS install depends first on standardizing the concept of a source wheel (or sdist 2.0). This is because, in my mind, the main two things you can do with a VCS install, from pip's point of view, is do an in place installation or create a source wheel. I also agree with Paul that we should not be adding new formats to PyPI that do not support static metadata. What metadata in specific makes sense for that particular format (like ABI doesn't make sense for a source wheel/sdist) is going to be format dependent, but there should never be a "run this bit of code to find out X" situation. The closest we should get is "that type of metadata doesn't make sense for X, you need to build X into a Y for it". PyPI needs static metadata [6], and static metadata makes other tools easier to operate as well. [1] I'm going to just call these VCS installs, but I mean any install that is not a fully formed Python package and is instead just an arbitrary collection of files and directories where we can have a minimal amount of control over the structure. [2] This goes back to my belief that one of the "original sins" of distutils and setuptools was blurring the lines between the different phases in the life cycle of a package. [3] Unlike Paul, I think a Source Wheel is actually a decent name for this concept. It's similar to .rpm and .src.rpm in that world, and I think it makes it more obvious that this item isn't an installable item in it's own right, that it exists in order to produce binary wheels. However, this concept is currently being handled by the sdist "format", however ambigiously defined that currently is. I also think it makes it a bit easier to get rid of the ambigous "package", since we can just call them all "wheels" which is easier to say than "distribution". [4] Even though I want to reduce the number of paths we take, I don't think we'll ever be able to reasonably get rid of the inplace installation path. There will probably need to be some back and forth about exactly which parts of the inplace install the installer should be responsible for and which parts the build tool is responsible for. [5] Mandatory in the sense of, if you're going to upload any files to PyPI you must upload one of these, not mandatory in the absolute sense. In a future world, ideally the only thing people will need to upload is a source wheel and we'll have a build farm that will take that and automatically produce binary wheels from them. [6] In reality, we don't have static metadata today, and we get by. This is mostly because we force uploads to include static metadata alongside the upload and we present that instead. In the future I want to move us to a situation where you *just* upload the file, and PyPI inspects the file for all of the metadata it needs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA _______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
