Re: [Distutils] distlib - installer support vs runtime support
On Mon, Mar 11, 2013 at 11:36 AM, Erik Bray wrote: > On Sat, Mar 9, 2013 at 9:33 PM, PJ Eby wrote: >> On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan wrote: >>> Longer term, something like the import engine PEP may let us implement a >>> cleaner solution. >> >> I've been giving the bootstrapping issue a bit more thought, though, >> and I think there's a way to take the pain out of multi-file >> bootstraps of things like pip and setuptools and whatnot. >> >> Suppose you make a .py file that contains (much like ez_setup.py) a >> list of files to download, along with their checksums. And suppose >> this .py file basically just has some code that checks a local cache >> directory for those files, adds them to sys.path if they're found, >> downloads them if they're not (validating the checksum), and then >> proceeds to import a specific entry point and run it? > > I've been thinking about something like this lately too. A simple > script like your proposed pystart could be used to generate these > files. I've gone one further and considered a format that includes > dependencies (or at the very least install-time dependencies right in > the .py file itself as a base64 string. This would work fine at least > for small dependencies--no downloads necessary and you're guaranteed > to get the right version that actually installs the damn package. You could, but it makes the initial download bigger and adds more moving parts. If you have to download some of them, might as well do all of them. (Besides, the approach I outlined allows for sharing distributions and avoiding repeated downloads.) > For packages using setuptools it's also possible to just include a > dependency as an egg or tar.gz right in your source dist and add > something like "[easy_install]\nfind_links = ." to your setup.cfg and > it'll go. But I'm not sure that's exactly the use case you're after > here. No. A particular goal behind this idea is that a pystart script should be cross-platform and ideally cross-Python-version as well, using environment markers to determine what should be downloaded. So, syntactically, the script would be written so that it runs on as many Python versions as possible, which shouldn't be too hard since it will basically be just looking for a series of files, downloading and hashing them. You'd have one function that does a "check for this file, download and extract if needed (w/hash validation), add to sys.path", and a bunch of if/then blocks calling that function with different name/url/hash triplets according to the current platform/python version markers. Then the bottom of the script would import the script entry point and run it. (And the top would have a preamble to import stuff from the right places based on Python version.) Alternatively, for a simpler code generation model, it could just stick everything in a giant data structure at the top, and append standard code that checks the environment markers. But I'd rather not have to include full environment marker interpretation, so translating the markers to Python seems advisable, even if it's as lambdas in the data structure. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
On Sat, Mar 9, 2013 at 9:33 PM, PJ Eby wrote: > On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan wrote: >> Longer term, something like the import engine PEP may let us implement a >> cleaner solution. > > I've been giving the bootstrapping issue a bit more thought, though, > and I think there's a way to take the pain out of multi-file > bootstraps of things like pip and setuptools and whatnot. > > Suppose you make a .py file that contains (much like ez_setup.py) a > list of files to download, along with their checksums. And suppose > this .py file basically just has some code that checks a local cache > directory for those files, adds them to sys.path if they're found, > downloads them if they're not (validating the checksum), and then > proceeds to import a specific entry point and run it? I've been thinking about something like this lately too. A simple script like your proposed pystart could be used to generate these files. I've gone one further and considered a format that includes dependencies (or at the very least install-time dependencies right in the .py file itself as a base64 string. This would work fine at least for small dependencies--no downloads necessary and you're guaranteed to get the right version that actually installs the damn package. For packages using setuptools it's also possible to just include a dependency as an egg or tar.gz right in your source dist and add something like "[easy_install]\nfind_links = ." to your setup.cfg and it'll go. But I'm not sure that's exactly the use case you're after here. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
The pkg_resources script entry point is there so the right eggs can be added to sys.path based on solving dependencies for the invoked package. On Mar 10, 2013 7:30 AM, "Paul Moore" wrote: > > On 10 March 2013 00:15, Vinay Sajip wrote: > > Paul Moore gmail.com> writes: > > > >> Would it be worth considering splitting distlib into two separate > >> parts - one that is intended solely for writers of installers and > >> similar tools, and another for "runtime support" functions that end > >> users would use? It may not be a practical thing to achieve, but it > >> would be worth at least understanding the trade-offs involved. > > > > While this could be done, it would not exactly be elegant, and IMO it would be > > the wrong way to address the valid concerns you mention. It would make more > > sense for pip to *contain* a specific version of distlib for its use (e.g. as a > > .zip) so that it never worries about conflicts with another copy. This is the > > approach that Django takes and it seems to work reasonably well for that > > project. > > Thanks for the comments. It looks like pip will indeed contain a copy > of distlib, at least for now. And I think you're right, that is the > best solution there. > > My other interest was with regard to virtualenv. Here, the particular > "runtime support" issue that bothers me is the way that setuptools > wrapper scripts use entry points. As a result, something like > nosetests.exe will not work without setuptools being present, simply > because it looks up the code to run using the entry point mechanisms > (the actual code itself does not need setuptools). So virtualenv > pretty much *has* to preinstall setuptools (or at least > pkg_resources), as pip uses exe-wrappers, and those won't use an > embedded copy. > > But looking at the code generated by distlib's script wrappers, I see > that it does not use the exports functionality of distlib, and as a > result distlib-generated wrappers can be used without distlib being > present. So my apologies here - it looks like my concern was > unfounded. > > Paul. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
On 10 March 2013 00:15, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> Would it be worth considering splitting distlib into two separate >> parts - one that is intended solely for writers of installers and >> similar tools, and another for "runtime support" functions that end >> users would use? It may not be a practical thing to achieve, but it >> would be worth at least understanding the trade-offs involved. > > While this could be done, it would not exactly be elegant, and IMO it would be > the wrong way to address the valid concerns you mention. It would make more > sense for pip to *contain* a specific version of distlib for its use (e.g. as > a > .zip) so that it never worries about conflicts with another copy. This is the > approach that Django takes and it seems to work reasonably well for that > project. Thanks for the comments. It looks like pip will indeed contain a copy of distlib, at least for now. And I think you're right, that is the best solution there. My other interest was with regard to virtualenv. Here, the particular "runtime support" issue that bothers me is the way that setuptools wrapper scripts use entry points. As a result, something like nosetests.exe will not work without setuptools being present, simply because it looks up the code to run using the entry point mechanisms (the actual code itself does not need setuptools). So virtualenv pretty much *has* to preinstall setuptools (or at least pkg_resources), as pip uses exe-wrappers, and those won't use an embedded copy. But looking at the code generated by distlib's script wrappers, I see that it does not use the exports functionality of distlib, and as a result distlib-generated wrappers can be used without distlib being present. So my apologies here - it looks like my concern was unfounded. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
On Sat, Mar 9, 2013 at 8:14 PM, Nick Coghlan wrote: > > On 10 Mar 2013 10:16, "Vinay Sajip" wrote: >> >> Paul Moore gmail.com> writes: >> >> > Would it be worth considering splitting distlib into two separate >> > parts - one that is intended solely for writers of installers and >> > similar tools, and another for "runtime support" functions that end >> > users would use? It may not be a practical thing to achieve, but it >> > would be worth at least understanding the trade-offs involved. >> >> While this could be done, it would not exactly be elegant, and IMO it >> would be >> the wrong way to address the valid concerns you mention. It would make >> more >> sense for pip to *contain* a specific version of distlib for its use (e.g. >> as a >> .zip) so that it never worries about conflicts with another copy. This is >> the >> approach that Django takes and it seems to work reasonably well for that >> project. >> >> Re. the size of distlib - it's larger than pip, because pip relies on a >> external dependency (setuptools/distribute) to do a lot of its work, >> whereas >> distlib is self-contained. So, direct size comparisons can be misleading >> (e.g. >> distlib contains a _backport package for 2.6 support, which is not tiny; >> distlib's tests contain tests for the backports plus a complete copy of >> unittest2, again for 2.6 support). >> >> The concerns about stability (in terms of API stability, as well as the >> presence of bugs) are more valid than size. Given distlib's youth, we can >> expect to see some API changes, as well as to see some bugs flushed out of >> the >> woodwork. (Obviously, I would aim to keep API changes to a minimum.) >> >> A good level of stability is generally achieved after a period of usage in >> anger followed by feedback based on that usage. Until then, the test >> suite, >> coverage results and docs will need to be used to give an indication of >> quality. > > pip vendoring its own copy of distlib sounds like the best workaround for > now, as it addresses both the bootstrapping problem and the API stability > question. > > Longer term, something like the import engine PEP may let us implement a > cleaner solution. I've been giving the bootstrapping issue a bit more thought, though, and I think there's a way to take the pain out of multi-file bootstraps of things like pip and setuptools and whatnot. Suppose you make a .py file that contains (much like ez_setup.py) a list of files to download, along with their checksums. And suppose this .py file basically just has some code that checks a local cache directory for those files, adds them to sys.path if they're found, downloads them if they're not (validating the checksum), and then proceeds to import a specific entry point and run it? At that point, you could distribute a Python application (like pip) "in" a single .py file. The file is for a specific version and a specific set of dependencies, but if someone wants to update it, they just download the new .py file and discard the old one. (Technically, it needn't have a .py extension, and for Windows you might want to ship it as a .exe+"-script.py/pyw" wrapper pair.) The really interesting bit is that you could generate the .py part using nothing more than a Metadata 2.0 file for the target app, access to the /simple index, and a specification of which script/entry-point the .py file is supposed to represent. We could call this concept PyStart, as it's a little bit like Java WebStart. It would work equally well with wheels or eggs, though with wheels you'd need to unpack any "impure" ones to a subdirectory. The main idea, though, is that if properly done, the PyStart script for an application wouldn't need to understand version parsing or comparison, PyPI APIs, environment markers, or any of that crud. (It *might* need to understand platform tags to some extent, if "impure" code is involved.) Heck, it wouldn't even care about SSL cert verification, as it'll be relying on its hardcoded hashes for verification purposes. So as long as you download the PyStart script from a trusted source, you're ready to go. (This is an important requirement for setuptools going forward, because it may have to depend on more than one other project (e.g. Requests + SSL backport) in order to implement full SSL security.) This is far from a finished idea, but I think it has some merit as an approach for distributing "zero-install" Python apps in general, while working around the hairy problem of things like, "so, to install pip, you need to have setuptools installed. But you can't use setuptools to install pip, because that wouldn't be secure. So, make sure you download *both* things by hand, oh, and don't forget to compile the SSL backport..." Thoughts? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
On 10 Mar 2013 10:16, "Vinay Sajip" wrote: > > Paul Moore gmail.com> writes: > > > Would it be worth considering splitting distlib into two separate > > parts - one that is intended solely for writers of installers and > > similar tools, and another for "runtime support" functions that end > > users would use? It may not be a practical thing to achieve, but it > > would be worth at least understanding the trade-offs involved. > > While this could be done, it would not exactly be elegant, and IMO it would be > the wrong way to address the valid concerns you mention. It would make more > sense for pip to *contain* a specific version of distlib for its use (e.g. as a > .zip) so that it never worries about conflicts with another copy. This is the > approach that Django takes and it seems to work reasonably well for that > project. > > Re. the size of distlib - it's larger than pip, because pip relies on a > external dependency (setuptools/distribute) to do a lot of its work, whereas > distlib is self-contained. So, direct size comparisons can be misleading (e.g. > distlib contains a _backport package for 2.6 support, which is not tiny; > distlib's tests contain tests for the backports plus a complete copy of > unittest2, again for 2.6 support). > > The concerns about stability (in terms of API stability, as well as the > presence of bugs) are more valid than size. Given distlib's youth, we can > expect to see some API changes, as well as to see some bugs flushed out of the > woodwork. (Obviously, I would aim to keep API changes to a minimum.) > > A good level of stability is generally achieved after a period of usage in > anger followed by feedback based on that usage. Until then, the test suite, > coverage results and docs will need to be used to give an indication of > quality. pip vendoring its own copy of distlib sounds like the best workaround for now, as it addresses both the bootstrapping problem and the API stability question. Longer term, something like the import engine PEP may let us implement a cleaner solution. Cheers, Nick. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
Paul Moore gmail.com> writes: > Would it be worth considering splitting distlib into two separate > parts - one that is intended solely for writers of installers and > similar tools, and another for "runtime support" functions that end > users would use? It may not be a practical thing to achieve, but it > would be worth at least understanding the trade-offs involved. While this could be done, it would not exactly be elegant, and IMO it would be the wrong way to address the valid concerns you mention. It would make more sense for pip to *contain* a specific version of distlib for its use (e.g. as a .zip) so that it never worries about conflicts with another copy. This is the approach that Django takes and it seems to work reasonably well for that project. Re. the size of distlib - it's larger than pip, because pip relies on a external dependency (setuptools/distribute) to do a lot of its work, whereas distlib is self-contained. So, direct size comparisons can be misleading (e.g. distlib contains a _backport package for 2.6 support, which is not tiny; distlib's tests contain tests for the backports plus a complete copy of unittest2, again for 2.6 support). The concerns about stability (in terms of API stability, as well as the presence of bugs) are more valid than size. Given distlib's youth, we can expect to see some API changes, as well as to see some bugs flushed out of the woodwork. (Obviously, I would aim to keep API changes to a minimum.) A good level of stability is generally achieved after a period of usage in anger followed by feedback based on that usage. Until then, the test suite, coverage results and docs will need to be used to give an indication of quality. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
On Sat, Mar 9, 2013 at 1:28 PM, Paul Moore wrote: > Would it be worth considering splitting distlib into two separate > parts - one that is intended solely for writers of installers and > similar tools, and another for "runtime support" functions that end > users would use? It may not be a practical thing to achieve, but it > would be worth at least understanding the trade-offs involved. > > Paul. > > PS The same issue exists in setuptools, but as far as I am aware, > setuptools was deliberately designed to provide both the installation > tools and runtime support in the one package. (And IIUC, even with > setuptools, pkg_resources is the only component that includes runtime > support, so in theory it is possible to split the two parts up). Yeah, the goal in setuptools case was to bootstrap dependency handling by only installing one thing. A major headache that's hitting me right now about that is that it's basically impossible to do that and provide SSL verification support at the same time, for versions of Python <2.6. So I'm pretty soon going to have to face this challenge also, in order to get to a "secure by default" setuptools for Pythons 2.3-2.5. Ironically, setuptools doesn't have any problems updating itself or its dependencies, though (if it had them). For example, if setuptools depended on one version of distlib, and the user requested installation of another version, this wouldn't affect setuptools at all, because the require()s baked into its scripts would still refer to the same distlib egg as before. Under the pip+virtualenv paradigm, though, there aren't any eggs, so you can't pin a single project to a dependency version, without giving pip its own virtualenv. And I've tried to think of a workaround to suggest, but the challenge is that they all boil down to something that requires you to already have distlib, in order to manage its own installation. ;-) Probably the best way to work around it is to just give pip its own virtualenv -- in the form of an executable zip file. That is, install pip and its dependencies to a single directory, add a __main__.py, and zip up the whole thing with a #!python header. Voila -- a self-contained application you can drop on your path, and update by just downloading a new copy of. Of course, that only works if pip is an application, rather than a library. Setuptools can't take this tack because it wants to be importable. But if pip only needs to be runnable, not importable, you could bundle *all* your dependencies in a single file, as long as they're all pure Python. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distlib - installer support vs runtime support
It would be great to maintain and install pkg_resources separately. The idea has come up before, including the idea of just putting pkg_resources in the system library without the rest of setuptools (it will stay on pypi now - pip install pkg_resources) On Mar 9, 2013 1:29 PM, "Paul Moore" wrote: > As part of a discussion on a pip issue, it was noted that if pip > depends on distlib for installer capabilities (e.g., locators, wheel > installation, version and requirement parsing and matching, etc) then > the user needs distlib installed as well as pip. That shouldn't be an > issue for an end user, any more than having pip itself installed would > be. > > However, there are some capabilites of distlib (notably the "exports" > functionality that replaces setuptools' entry points) that are > intended to be used at runtime, by the end user. (I suspect, but > haven't checked, that the exe wrapper functionality depends on exports > as well - it certainly does in setuptools). That means that the end > user *is* affected by the fact that pip depends on distlib. For > example, if the user needs a later version of distlib, it's quite > likely that he won't be able to "pip install" it, as pip typically has > trouble upgrading its own dependencies (for obvious reasons...) > > Would it be worth considering splitting distlib into two separate > parts - one that is intended solely for writers of installers and > similar tools, and another for "runtime support" functions that end > users would use? It may not be a practical thing to achieve, but it > would be worth at least understanding the trade-offs involved. > > Paul. > > PS The same issue exists in setuptools, but as far as I am aware, > setuptools was deliberately designed to provide both the installation > tools and runtime support in the one package. (And IIUC, even with > setuptools, pkg_resources is the only component that includes runtime > support, so in theory it is possible to split the two parts up). > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] distlib - installer support vs runtime support
As part of a discussion on a pip issue, it was noted that if pip depends on distlib for installer capabilities (e.g., locators, wheel installation, version and requirement parsing and matching, etc) then the user needs distlib installed as well as pip. That shouldn't be an issue for an end user, any more than having pip itself installed would be. However, there are some capabilites of distlib (notably the "exports" functionality that replaces setuptools' entry points) that are intended to be used at runtime, by the end user. (I suspect, but haven't checked, that the exe wrapper functionality depends on exports as well - it certainly does in setuptools). That means that the end user *is* affected by the fact that pip depends on distlib. For example, if the user needs a later version of distlib, it's quite likely that he won't be able to "pip install" it, as pip typically has trouble upgrading its own dependencies (for obvious reasons...) Would it be worth considering splitting distlib into two separate parts - one that is intended solely for writers of installers and similar tools, and another for "runtime support" functions that end users would use? It may not be a practical thing to achieve, but it would be worth at least understanding the trade-offs involved. Paul. PS The same issue exists in setuptools, but as far as I am aware, setuptools was deliberately designed to provide both the installation tools and runtime support in the one package. (And IIUC, even with setuptools, pkg_resources is the only component that includes runtime support, so in theory it is possible to split the two parts up). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
