Re: [Distutils] PEP 517: Build system API
On 25 November 2016 at 14:04, Nick Coghlan wrote: .. > > The bad reputation of ".pth" doesn't generally stem from their normal > usage (i.e. just listing absolute or relative directory names that the > import system will then append to __path__). > > Rather, it stems from this little aspect: "Lines starting with import > (followed by space or tab) are executed." (from > https://docs.python.org/3/library/site.html ) I think its also worth exploring a targeted, modern namespace aware replacement. That is - there are two, related, use cases for .pth files vis-a-vis package installation. One is legacy namespace packages, which AFAICT are still in use - the migration is "complicated". The second is arguable a variant of that same thing: putting the current working dir into the PYTHONPATH without putting it in PYTHONPATH. The former case may be sufficiently covered by (forget the pep #) that we don't need to do anything, and the latter is certainly something that we should be able to deliver without needing the turing complete capability that you're suggesting. Something data driven rather than code driven. -Rob ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On 25 November 2016 at 09:10, Paul Moore wrote: > Honestly, I don't see what's so bad about pth files. They are a > standard supported Python approach. Maybe setuptools' use of them is > messy? I recall it was possible to end up with a lot of clutter, but > that was going back to the egg days, I'm not sure if it's like that > any more. Why not just have a single pth file, maintained by the build > tool, for all editable installs? Most users would then only have one > or two pth files The bad reputation of ".pth" doesn't generally stem from their normal usage (i.e. just listing absolute or relative directory names that the import system will then append to __path__). Rather, it stems from this little aspect: "Lines starting with import (followed by space or tab) are executed." (from https://docs.python.org/3/library/site.html ) As such, rather than purely being a way of extending "__path__" with additional directories, path files are *also* a way of programmatically customising a particular Python installation, without risking filename collisions the way sitecustomize and usercustomize do. Since setuptools is the first or only encounter most folks have with ".pth" files, and it makes use of such lines to execute arbitrary code at interpreter startup, the targeted criticism of "the 'import line' feature of path files and the specific ways that setuptools uses that feature are hard to debug when they break" morphed over time into the blanket criticism of "path files are bad". The main problem with fixing that is that folks really do rely on the auto-execution feature, sometimes even without knowing it (e.g. because setuptools uses it implicitly). Therefore, in order to deprecate and remove the magic import lines, we'd need to do something like adding an "__autorun__" directory to site-packages and user site-packages as a long term replacement for: - sitecustomize.py - usercustomize.py - executable import lines in path files Supporting such a subdirectory in older Python versions could then be bootstrapped via an "autorun.pth" file that provided the same start-up logic that 3.7+ did. I suspect this would actually be an improvement from a security awareness perspective as well, as it would make it much clearer that enabling implicit code execution at Python interpreter start-up is trivial if you have write access to the site-packages or user site-packages directory. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
Thomas Kluyver wrote: - There's a feature called NTFS Junction Points, which is supposed to be like symlinks, but only for directories. Things might have changed, but last time I played with junction points they seemed rather dangerous, because if you deleted one using the GUI it deleted the contents of the directory it was linked to. And it appeared indistinguishible from a normal folder in the GUI, so you got no warning that was about to happen. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On 24 November 2016 at 21:51, Thomas Kluyver wrote: > On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote: >> Just curious - how does flit handle Windows for this? Symlinks aren't >> really an option there (you need elevation to create a symlink). >> Paul > > It largely doesn't at present; it started out as a personal tool for me, > and I mostly use Linux. I have a few ideas about how to deal with > Windows, though: > > - There's a feature called NTFS Junction Points, which is supposed to be > like symlinks, but only for directories. I believe these can be created > by regular users without elevated permissions. It might be possible to > use this for packages, albeit not for single file directories. > - Create a script/tool to give a user the permission bit that allows > creating symlinks, so users who know what they're doing can enable it. > Maybe there's some reason it's admin-only by default, though? It's safe > enough for normal users on Unix, but I don't know how different Windows > is. However you do it, it needs elevation, which is going to be a problem. I don't think symlinks per se are any worse on Windows (apart from the unfamiliarity aspect, which also applies to junction points, which may mean tools don't handle them gracefully). But because you can't use them without elevation/permission fiddling, they simply aren't an end-user concept on Windows. > - Run a daemon to watch package folders and reinstall on any changes, > something like Nathaniel mentioned with 'pip watch'. I probably wouldn't > do this as part of flit, but I'd be happy to see it as a separate tool. > As Nathaniel pointed out, this can actually support more stuff than > setuptools editable installs, or my symlinks, because it can re-run > build steps. That sounds like it could also have some issues making it work cleanly on Windows (would it leave a console window open?). Honestly, I don't see what's so bad about pth files. They are a standard supported Python approach. Maybe setuptools' use of them is messy? I recall it was possible to end up with a lot of clutter, but that was going back to the egg days, I'm not sure if it's like that any more. Why not just have a single pth file, maintained by the build tool, for all editable installs? Most users would then only have one or two pth files. Anyway, this is OT, and I'm sure you'll come up with something appropriate as and when the need arises. As I say, I was mostly just curious. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
I've made PRs against PEP 517 for: Underscore to dash in build-backend: https://github.com/python/peps/pull/139 1a: Add a user parameter to the install_editable hook https://github.com/python/peps/pull/140 OR: 1c: Get rid of the install_editable hook https://github.com/python/peps/pull/141 I'm leaning towards 1c; it can always be standardised in a separate PEP, and an optional hook added back. Thomas On Thu, Nov 24, 2016, at 09:06 AM, Paul Moore wrote: > On 24 November 2016 at 00:32, Nathaniel Smith wrote: > > Also note that just like we decided to split the basic pyproject.toml > > proposal (now PEP 518) from the build system interface proposal (now > > PEP 517), it might (probably) makes sense to split the editable > > install part of the build system interface proposal from the rest, > > just in the interests of making incremental progress. > > This would be my preference. I'm assuming (again, incremental > progress) that "pip install -e" will still work as at present for > projects that don't use the new infrastructure, so it's hardly an > urgent need. > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote: > Just curious - how does flit handle Windows for this? Symlinks aren't > really an option there (you need elevation to create a symlink). > Paul It largely doesn't at present; it started out as a personal tool for me, and I mostly use Linux. I have a few ideas about how to deal with Windows, though: - There's a feature called NTFS Junction Points, which is supposed to be like symlinks, but only for directories. I believe these can be created by regular users without elevated permissions. It might be possible to use this for packages, albeit not for single file directories. - Create a script/tool to give a user the permission bit that allows creating symlinks, so users who know what they're doing can enable it. Maybe there's some reason it's admin-only by default, though? It's safe enough for normal users on Unix, but I don't know how different Windows is. - Run a daemon to watch package folders and reinstall on any changes, something like Nathaniel mentioned with 'pip watch'. I probably wouldn't do this as part of flit, but I'd be happy to see it as a separate tool. As Nathaniel pointed out, this can actually support more stuff than setuptools editable installs, or my symlinks, because it can re-run build steps. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On 24 November 2016 at 19:50, Thomas Kluyver wrote: > I hate the thing setuptools does with .pth files all over the place, so the > equivalent operation in flit symlinks packages to site-packages. Just curious - how does flit handle Windows for this? Symlinks aren't really an option there (you need elevation to create a symlink). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On Thu, Nov 24, 2016, at 07:50 PM, Thomas Kluyver wrote: > I made a PR to flit to handle this case better in uninstallation. I meant a PR to *pip*. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On Thu, Nov 24, 2016, at 12:23 AM, Daniel Holth wrote: > I wouldn't be afraid of editable installs. They are trivial and > involve building the package in place and putting a .pth file where it > will be noticed. Specify editable packages can't necessarily be > uninstalled in a standard way and you are done. > The bespoke build tool tells pip where the package root is (where .dist- > info will be written), usually . or ./src, then pip does .pth. The way it's specified at present is for pip to ask the build tool (setuptools, flit, etc.) to do an editable install by whatever means. I hate the thing setuptools does with .pth files all over the place, so the equivalent operation in flit symlinks packages to site-packages. I made a PR to flit to handle this case better in uninstallation. I think Nathaniel's right: editable installs are going to be a source of controversity and complexity. Let's drop that hook from the PEP for now (ie. 1c) - it's still useful without it, and the design of the build interface allows extra hooks to be specified and easily added in the future. Does anyone feel strongly that we should keep the editable install hook in the current PEP? If not, I'll make a PR removing it (and switching the underscore to dash, as that seems to be the consensus). I'm also happy to look into implementing PEPs 518 and 517 in pip, though it might be a week or two before I get time. Thanks, Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging multiple wheels in the same archive
On Nov 23, 2016 11:33 AM, "Brett Cannon" wrote: > > This then ties into Kenneth's pipfile idea he's working on as it then makes sense to make a wagon/wheelhouse for a lock file. To also tie into the container aspect, if you dev on Windows but deploy to Linux, this can allow for gathering your dependencies locally for Linux on your Windows box and then deploy the set as a unit to your server (something Steve Dower and I have thought about and why we support a lock file concept). > > And if we use zip files with no nesting then as long as it's only Python code you could use zipimporter on the bundle directly. The "only Python code" restriction pretty much rules this out as anything like a general solution though... If people are investigating this, though, pex should also be considered as a source of prior art / inspiration: https://pex.readthedocs.io/en/stable/whatispex.html -n ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging multiple wheels in the same archive
On 24 November 2016 at 16:45, Nir Cohen wrote: > Well, creating on Windows and deploying on Linux will only be possible if > the entire set of dependencies either have no C extensions or are manylinux1 > wheels.. but yeah, that's pretty much what we're doing right now with our > reference implementation. > > Regarding zipimporter, as far as I understand (correct me if I'm wrong) > there's no such a solution for wheels (i.e. you can't use zipimporter on a > zip of wheels) so does that means we'll have to package python files for all > dependencies directly in the archive? Right, there would be a couple of significant barriers to doing this in the general case: - firstly, wheels themselves are officially only a transport format, with direct imports being a matter of "we're not going to do anything to deliberately break the cases that work, but you're also on your own if anything goes wrong for any given use case": https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file - secondly, I don't think zipimporter handles archives-within-archives - it handles directories within archives, so it would require that the individual wheels by unpacked and the whole structure archived as one big directory tree Overall, it sounds to me more like the "archive an entire installed virtual environment" use case than it does the "transfer a collection of pre-built artifacts from point A to point B" use case (which, to be fair, is an interesting use case in its own right, its just a slightly different problem). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 517: Build system API
On 24 November 2016 at 00:32, Nathaniel Smith wrote: > Also note that just like we decided to split the basic pyproject.toml > proposal (now PEP 518) from the build system interface proposal (now > PEP 517), it might (probably) makes sense to split the editable > install part of the build system interface proposal from the rest, > just in the interests of making incremental progress. This would be my preference. I'm assuming (again, incremental progress) that "pip install -e" will still work as at present for projects that don't use the new infrastructure, so it's hardly an urgent need. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig