Re: [Distutils] PEP 517: Build system API

[Robert Collins]

On 25 November 2016 at 14:04, Nick Coghlan  wrote:
..
>
> The bad reputation of ".pth" doesn't generally stem from their normal
> usage (i.e. just listing absolute or relative directory names that the
> import system will then append to __path__).
>
> Rather, it stems from this little aspect: "Lines starting with import
> (followed by space or tab) are executed." (from
> https://docs.python.org/3/library/site.html )

I think its also worth exploring a targeted, modern namespace aware replacement.

That is - there are two, related, use cases for .pth files vis-a-vis
package installation. One is legacy namespace packages, which AFAICT
are still in use - the migration is "complicated". The second is
arguable a variant of that same thing: putting the current working dir
into the PYTHONPATH without putting it in PYTHONPATH.

The former case may be sufficiently covered by (forget the pep #) that
we don't need to do anything, and the latter is certainly something
that we should be able to deliver without needing the turing complete
capability that you're suggesting. Something data driven rather than
code driven.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 25 November 2016 at 09:10, Paul Moore  wrote:
> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy? I recall it was possible to end up with a lot of clutter, but
> that was going back to the egg days, I'm not sure if it's like that
> any more. Why not just have a single pth file, maintained by the build
> tool, for all editable installs? Most users would then only have one
> or two pth files

The bad reputation of ".pth" doesn't generally stem from their normal
usage (i.e. just listing absolute or relative directory names that the
import system will then append to __path__).

Rather, it stems from this little aspect: "Lines starting with import
(followed by space or tab) are executed." (from
https://docs.python.org/3/library/site.html )

As such, rather than purely being a way of extending "__path__" with
additional directories, path files are *also* a way of
programmatically customising a particular Python installation, without
risking filename collisions the way sitecustomize and usercustomize
do.

Since setuptools is the first or only encounter most folks have with
".pth" files, and it makes use of such lines to execute arbitrary code
at interpreter startup, the targeted criticism of "the 'import line'
feature of path files and the specific ways that setuptools uses that
feature are hard to debug when they break" morphed over time into the
blanket criticism of "path files are bad".

The main problem with fixing that is that folks really do rely on the
auto-execution feature, sometimes even without knowing it (e.g.
because setuptools uses it implicitly).

Therefore, in order to deprecate and remove the magic import lines,
we'd need to do something like adding an "__autorun__" directory to
site-packages and user site-packages as a long term replacement for:

- sitecustomize.py
- usercustomize.py
- executable import lines in path files

Supporting such a subdirectory in older Python versions could then be
bootstrapped via an "autorun.pth" file that provided the same start-up
logic that 3.7+ did.

I suspect this would actually be an improvement from a security
awareness perspective as well, as it would make it much clearer that
enabling implicit code execution at Python interpreter start-up is
trivial if you have write access to the site-packages or user
site-packages directory.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Greg Ewing]

Thomas Kluyver wrote:

- There's a feature called NTFS Junction Points, which is supposed to be
like symlinks, but only for directories.


Things might have changed, but last time I played with
junction points they seemed rather dangerous, because
if you deleted one using the GUI it deleted the contents
of the directory it was linked to. And it appeared
indistinguishible from a normal folder in the GUI, so
you got no warning that was about to happen.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 21:51, Thomas Kluyver  wrote:
> On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote:
>> Just curious - how does flit handle Windows for this? Symlinks aren't
>> really an option there (you need elevation to create a symlink).
>> Paul
>
> It largely doesn't at present; it started out as a personal tool for me,
> and I mostly use Linux. I have a few ideas about how to deal with
> Windows, though:
>
> - There's a feature called NTFS Junction Points, which is supposed to be
> like symlinks, but only for directories. I believe these can be created
> by regular users without elevated permissions. It might be possible to
> use this for packages, albeit not for single file directories.
> - Create a script/tool to give a user the permission bit that allows
> creating symlinks, so users who know what they're doing can enable it.
> Maybe there's some reason it's admin-only by default, though? It's safe
> enough for normal users on Unix, but I don't know how different Windows
> is.

However you do it, it needs elevation, which is going to be a problem.
I don't think symlinks per se are any worse on Windows (apart from the
unfamiliarity aspect, which also applies to junction points, which may
mean tools don't handle them gracefully). But because you can't use
them without elevation/permission fiddling, they simply aren't an
end-user concept on Windows.

> - Run a daemon to watch package folders and reinstall on any changes,
> something like Nathaniel mentioned with 'pip watch'. I probably wouldn't
> do this as part of flit, but I'd be happy to see it as a separate tool.
> As Nathaniel pointed out, this can actually support more stuff than
> setuptools editable installs, or my symlinks, because it can re-run
> build steps.

That sounds like it could also have some issues making it work cleanly
on Windows (would it leave a console window open?).

Honestly, I don't see what's so bad about pth files. They are a
standard supported Python approach. Maybe setuptools' use of them is
messy? I recall it was possible to end up with a lot of clutter, but
that was going back to the egg days, I'm not sure if it's like that
any more. Why not just have a single pth file, maintained by the build
tool, for all editable installs? Most users would then only have one
or two pth files.

Anyway, this is OT, and I'm sure you'll come up with something
appropriate as and when the need arises. As I say, I was mostly just
curious.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

I've made PRs against PEP 517 for:

Underscore to dash in build-backend:
https://github.com/python/peps/pull/139

1a: Add a user parameter to the install_editable hook
https://github.com/python/peps/pull/140

OR:

1c: Get rid of the install_editable hook
https://github.com/python/peps/pull/141

I'm leaning towards 1c; it can always be standardised in a separate PEP,
and an optional hook added back.

Thomas

On Thu, Nov 24, 2016, at 09:06 AM, Paul Moore wrote:
> On 24 November 2016 at 00:32, Nathaniel Smith  wrote:
> > Also note that just like we decided to split the basic pyproject.toml
> > proposal (now PEP 518) from the build system interface proposal (now
> > PEP 517), it might (probably) makes sense to split the editable
> > install part of the build system interface proposal from the rest,
> > just in the interests of making incremental progress.
> 
> This would be my preference. I'm assuming (again, incremental
> progress) that "pip install -e" will still work as at present for
> projects that don't use the new infrastructure, so it's hardly an
> urgent need.
> 
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote:
> Just curious - how does flit handle Windows for this? Symlinks aren't
> really an option there (you need elevation to create a symlink).
> Paul

It largely doesn't at present; it started out as a personal tool for me,
and I mostly use Linux. I have a few ideas about how to deal with
Windows, though:

- There's a feature called NTFS Junction Points, which is supposed to be
like symlinks, but only for directories. I believe these can be created
by regular users without elevated permissions. It might be possible to
use this for packages, albeit not for single file directories.
- Create a script/tool to give a user the permission bit that allows
creating symlinks, so users who know what they're doing can enable it.
Maybe there's some reason it's admin-only by default, though? It's safe
enough for normal users on Unix, but I don't know how different Windows
is.
- Run a daemon to watch package folders and reinstall on any changes,
something like Nathaniel mentioned with 'pip watch'. I probably wouldn't
do this as part of flit, but I'd be happy to see it as a separate tool.
As Nathaniel pointed out, this can actually support more stuff than
setuptools editable installs, or my symlinks, because it can re-run
build steps.

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 19:50, Thomas Kluyver  wrote:
> I hate the thing setuptools does with .pth files all over the place, so the
> equivalent operation in flit symlinks packages to site-packages.

Just curious - how does flit handle Windows for this? Symlinks aren't
really an option there (you need elevation to create a symlink).
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 07:50 PM, Thomas Kluyver wrote:

> I made a PR to flit to handle this case better in uninstallation.

I meant a PR to *pip*.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging multiple wheels in the same archive

[Nathaniel Smith]

On Nov 23, 2016 11:33 AM, "Brett Cannon"  wrote:
>
> This then ties into Kenneth's pipfile idea he's working on as it then
makes sense to make a wagon/wheelhouse for a lock file. To also tie into
the container aspect, if you dev on Windows but deploy to Linux, this can
allow for gathering your dependencies locally for Linux on your Windows box
and then deploy the set as a unit to your server (something Steve Dower and
I have thought about and why we support a lock file concept).
>
> And if we use zip files with no nesting then as long as it's only Python
code you could use zipimporter on the bundle directly.

The "only Python code" restriction pretty much rules this out as anything
like a general solution though...

If people are investigating this, though, pex should also be considered as
a source of prior art / inspiration:
  https://pex.readthedocs.io/en/stable/whatispex.html

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Packaging multiple wheels in the same archive

[Nick Coghlan]

On 24 November 2016 at 16:45, Nir Cohen  wrote:
> Well, creating on Windows and deploying on Linux will only be possible if
> the entire set of dependencies either have no C extensions or are manylinux1
> wheels.. but yeah, that's pretty much what we're doing right now with our
> reference implementation.
>
> Regarding zipimporter, as far as I understand (correct me if I'm wrong)
> there's no such a solution for wheels (i.e. you can't use zipimporter on a
> zip of wheels) so does that means we'll have to package python files for all
> dependencies directly in the archive?

Right, there would be a couple of significant barriers to doing this
in the general case:

- firstly, wheels themselves are officially only a transport format,
with direct imports being a matter of "we're not going to do anything
to deliberately break the cases that work, but you're also on your own
if anything goes wrong for any given use case":
https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file
- secondly, I don't think zipimporter handles archives-within-archives
- it handles directories within archives, so it would require that the
individual wheels by unpacked and the whole structure archived as one
big directory tree

Overall, it sounds to me more like the "archive an entire installed
virtual environment" use case than it does the "transfer a collection
of pre-built artifacts from point A to point B" use case (which, to be
fair, is an interesting use case in its own right, its just a slightly
different problem).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 00:32, Nathaniel Smith  wrote:
> Also note that just like we decided to split the basic pyproject.toml
> proposal (now PEP 518) from the build system interface proposal (now
> PEP 517), it might (probably) makes sense to split the editable
> install part of the build system interface proposal from the rest,
> just in the interests of making incremental progress.

This would be my preference. I'm assuming (again, incremental
progress) that "pip install -e" will still work as at present for
projects that don't use the new infrastructure, so it's hardly an
urgent need.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig