Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 29 November 2016 at 21:41, Thomas Kluyver  wrote:
> On Thu, Nov 24, 2016, at 11:10 PM, Paul Moore wrote:
> But even without that, they're not ideal. I routinely have development
> installs of quite a lot of different packages. If each one of those is
> an entry on sys.path from a .pth file, there's a performance penalty, as
> every 'import qux' has to stat() in dozens of directories which don't
> contain qux at all:
>
> stat('.../foo/qux.py')
> stat('.../foo/qux/__init__.py')
> stat('.../foo/qux.so')
> stat('.../bar/qux.py')
> stat('.../bar/qux/__init__.py')
> ...
> stat('.../qux/qux.py')   # At last!

The Python 3 import system (and the importlib2 backport) cache
directory listings rather than making repeated stat calls, so the
performance penalty of longer paths is dramatically reduced relative
to the old "n stat calls per directory" approach. (For NFS based
imports with network round trips involved, we're talking multiple
orders of magnitude reductions in application start-up times)

> If you put tooling scripts/modules in the root of your project
> directory, it also increases the chance of name clashes.
>
> Maybe we should look into something more like symlinks for doing
> editable installs, so a file called '.../site-packages/qux.pylink' would
> contain the path to the qux module/package. This would only be read on
> 'import qux', so it wouldn't affect other imports at all.

The main challenge is to do this kind of thing without breaking PEP
376 (the directory based installation metadata layout), and without
coupling the adoption period to CPython adoption cycles (where 2.7
hasn't even completely supplanted 2.6, let alone the 3.x series taking
over from 2.x).

And executable .pth file *could* be used to bootstrap a new system,
but then the interaction with setuptools may get even more complicated
due to relative import orders (as far as I know the order of
processing for pth files is entirely arbitrary).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 11:10 PM, Paul Moore wrote:
> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy? I recall it was possible to end up with a lot of clutter, but
> that was going back to the egg days, I'm not sure if it's like that
> any more. Why not just have a single pth file, maintained by the build
> tool, for all editable installs? Most users would then only have one
> or two pth files.

They would definitely be a lot more tolerable if it wasn't for
setuptools' use of the execution loophole, and indeed that misfeature as
a whole.

But even without that, they're not ideal. I routinely have development
installs of quite a lot of different packages. If each one of those is
an entry on sys.path from a .pth file, there's a performance penalty, as
every 'import qux' has to stat() in dozens of directories which don't
contain qux at all:

stat('.../foo/qux.py')
stat('.../foo/qux/__init__.py')
stat('.../foo/qux.so')
stat('.../bar/qux.py')
stat('.../bar/qux/__init__.py')
...
stat('.../qux/qux.py')   # At last!

If you put tooling scripts/modules in the root of your project
directory, it also increases the chance of name clashes.

Maybe we should look into something more like symlinks for doing
editable installs, so a file called '.../site-packages/qux.pylink' would
contain the path to the qux module/package. This would only be read on
'import qux', so it wouldn't affect other imports at all.

Brett has merged my PR for option 1c, so PEP 517 no longer specifies a
hook for editable installs. If we need extra hooks to support editable
installs, we can specify them separately in a later PEP.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Robert Collins]

On 29 November 2016 at 07:23, Daniel Holth  wrote:
> Here is the design. The build tool provides the install tool with a
> directory, or if you want to get fancy, a list of directories, that should
> be added to PYTHONPATH. The build tool builds into that path. The install
> tool makes sure Python can find it. This will make anyone who uses
> setuptools happy.
>
> So PEP 517 does not need an editable install hook to support editable
> installs. It can just expose the equivalent of setuptools'
> package_dir={'':'src'}, and include a 'build in place' hook, and the feature
> is preserved.
>
> In this way setuptools users can be happy, and only users of build systems
> that can't work this way are waiting for a more complicated design.

Thats a possible route; note that some situations don't work with
in-place builds, and I don't see any reason to hardcode in-place
builds as the thing; this is why I was trying to punt on it, because
-e is actually somewhat deep in its own right.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 29 November 2016 at 04:23, Daniel Holth  wrote:
> Here is the design. The build tool provides the install tool with a
> directory, or if you want to get fancy, a list of directories, that should
> be added to PYTHONPATH. The build tool builds into that path. The install
> tool makes sure Python can find it. This will make anyone who uses
> setuptools happy.
>
> So PEP 517 does not need an editable install hook to support editable
> installs. It can just expose the equivalent of setuptools'
> package_dir={'':'src'}, and include a 'build in place' hook, and the feature
> is preserved.
>
> In this way setuptools users can be happy, and only users of build systems
> that can't work this way are waiting for a more complicated design.

I think this would also help resolve one of the limitations of PEP 517
as it currently stands, which is the differences between
environment-per-build and shared-environment, as the former is what
you want for releases, but the latter is often what you want locally
(e.g. to regenerate cffi and Cython bindings, or to ensure
dependencies and the project itself are importable).

"local_build" might be a better name for that than "build_in_place"
though, and the suggestion of "Sort out release builds first, tackle
local builds in a follow-up PEP" still has merit (since the use cases
and trade-offs for the latter case are genuinely different from those
for builds that are expected to be published to an index server).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Daniel Holth]

Here is the design. The build tool provides the install tool with a
directory, or if you want to get fancy, a list of directories, that should
be added to PYTHONPATH. The build tool builds into that path. The install
tool makes sure Python can find it. This will make anyone who uses
setuptools happy.

So PEP 517 does not need an editable install hook to support editable
installs. It can just expose the equivalent of setuptools'
package_dir={'':'src'}, and include a 'build in place' hook, and the
feature is preserved.

In this way setuptools users can be happy, and only users of build systems
that can't work this way are waiting for a more complicated design.

On Mon, Nov 28, 2016 at 1:01 PM Paul Moore  wrote:

> On 28 November 2016 at 17:53, Chris Barker  wrote:
> >> Why not just have a single pth file, maintained by the build
> >> tool, for all editable installs?
> >
> > shouldn't that be maintained by the install tool? i.e. pip -- the whole
> idea
> > is that the install tool is different than the built tool, yes? and
> adding a
> > package in editable mode is an installation job, not a build job.
> >
> > Also -- the idea here is that pip will know it's installed so it can
> > upgrade, de-install, etc, so it really is pip's job to maintain the
> > "editable_install.pth" file.
>
> Sorry - I was confusing "build tool" vs "install tool" here. Not
> intentionally, but the confusion is real. Setuptools is a build tool,
> and yet (currently) handles editable installs. So IMO, part of
> finalising editable install support would be thrashing out which
> aspects of the process are the responsibility of the build tool, and
> which the install tool. That's a non-trivial design job, so in the
> interests of keeping things moving, it seems to me that "defer a
> decision for now" remains the right decision here. Not to claim that
> editable installs aren't important, simply to avoid having all of the
> rest of the proposal stalled while we sort out the editable design.
>
> Paul
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Mon, Nov 28, 2016 at 10:01 AM, Paul Moore  wrote:

> On 28 November 2016 at 17:53, Chris Barker  wrote:
> >> Why not just have a single pth file, maintained by the build
> >> tool, for all editable installs?
> >
> > shouldn't that be maintained by the install tool? i.e. pip -- the whole
> idea
> > is that the install tool is different than the built tool, yes? and
> adding a
> > package in editable mode is an installation job, not a build job.
> >
> > Also -- the idea here is that pip will know it's installed so it can
> > upgrade, de-install, etc, so it really is pip's job to maintain the
> > "editable_install.pth" file.
>
> Sorry - I was confusing "build tool" vs "install tool" here. Not
> intentionally, but the confusion is real.


it sure is!


> Setuptools is a build tool,
> and yet (currently) handles editable installs.


setuptools is a build tool, and an install tool, and a runtime resource
manager, and a package manger / install tool -- source of lots of
confusion, and what we are trying to get away from, yes? ;-)

So IMO, part of
> finalising editable install support would be thrashing out which
> aspects of the process are the responsibility of the build tool, and
> which the install tool. That's a non-trivial design job, so in the
> interests of keeping things moving, it seems to me that "defer a
> decision for now" remains the right decision here.


fair enough.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 28 November 2016 at 17:53, Chris Barker  wrote:
>> Why not just have a single pth file, maintained by the build
>> tool, for all editable installs?
>
> shouldn't that be maintained by the install tool? i.e. pip -- the whole idea
> is that the install tool is different than the built tool, yes? and adding a
> package in editable mode is an installation job, not a build job.
>
> Also -- the idea here is that pip will know it's installed so it can
> upgrade, de-install, etc, so it really is pip's job to maintain the
> "editable_install.pth" file.

Sorry - I was confusing "build tool" vs "install tool" here. Not
intentionally, but the confusion is real. Setuptools is a build tool,
and yet (currently) handles editable installs. So IMO, part of
finalising editable install support would be thrashing out which
aspects of the process are the responsibility of the build tool, and
which the install tool. That's a non-trivial design job, so in the
interests of keeping things moving, it seems to me that "defer a
decision for now" remains the right decision here. Not to claim that
editable installs aren't important, simply to avoid having all of the
rest of the proposal stalled while we sort out the editable design.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Thu, Nov 24, 2016 at 3:10 PM, Paul Moore  wrote:

> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy?


exactly.

The fact that setuptools over-uses (abuses?) pth files doesn't mean that
they can't be used reasonably.

and given the differences of filesystems, it's a not-too-bad way to
simulate symbolic links :-)

Why not just have a single pth file, maintained by the build
> tool, for all editable installs?


shouldn't that be maintained by the install tool? i.e. pip -- the whole
idea is that the install tool is different than the built tool, yes? and
adding a package in editable mode is an installation job, not a build job.

Also -- the idea here is that pip will know it's installed so it can
upgrade, de-install, etc, so it really is pip's job to maintain the
"editable_install.pth" file.

Most users would then only have one
> or two pth files.
>

let's us make sure we don't end up with the easy_install.pth nightmare!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Wed, Nov 23, 2016 at 4:32 PM, Nathaniel Smith  wrote:

> On Wed, Nov 23, 2016 at 3:14 PM, Chris Barker 
> wrote:
>


> > Please, please, let's figure SOMETHING our here - editable installs (or
> > "develop" installs) are a critical tool. Frankly, I don't know how anyone
> > can develop a package without them.
>
> Would a 'pip watch' command that watched your source tree and
> rebuilt/reinstalled it whenever you edited a file work for you?


probably, yes -- though that seems pretty inefficient -- I suppose ti could
be smart enough to only update the changed file(s), which wouldn't be too
bad.

as for rebuild -- setuptools' develop mode doesn't rebuild anyway -- you
have to rebuild by hand if you change anything that isn't pure python --
which frankly works fine, as extensions need o be rebuild anyhow.

What
> do you do for conda packages?


develop mode is for, well, developing -- so no need for a conda package,
you are working from source be definition, and conda is a packaging system,
not a build system (i.e. for python packages, conda build usually calls
setuptools (ideally via pip) to do the building anyway)

Does conda have an editable install
> equivalent?
>

despite what I said above, conda does have a develop command:

http://conda.pydata.org/docs/commands/build/conda-develop.html

It's needed because conda does some trickery with editing relative paths
for linking shared libs at install time. If you use setuptools' develop
mode with an extension module, it may find the wrong version of libs at run
time.

I honestly don't know how well it works -- I built my own kludge for my
stuff before it was ready to go:

https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/re_link_for_anaconda.py

And this is a Mac only solution -- linking happens differently on Windows,
such that I think this is a non-issue, and not sure about Linux -- we are
deploying with conda on Linux, but I don't think anyone is developing on
Linux.


>  they make it harder
> to improve pip's infrastructure around upgrading, because upgrading
> requires pip to have a solid understanding of exactly what's
> installed. So it's absolutely possible to have some useful hack, like
> we do now, but useful hacks by their nature are hard to standardize:
>

True -- but I'd still rather a useful hack than nothing

standardization means "this is the final solution to this problem, it
> won't need further tweaking", and editable installs really feel like
> they might benefit from further tweaking. Or something.
>

maybe an officially marked as preliminary part of the standard?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Wes Turner]

I suppose a __main__.py could also/instead be added as

- site.tools.__main__
  https://github.com/python/cpython/blob/master/Lib/site.py
  - site.__doc__
  - site._script()

- distutils.__main__
  https://github.com/python/cpython/tree/master/Lib/distutils/command

- setuptools.__main__
  https://github.com/pypa/setuptools/blob/master/setup.py

   - _gen_console_scripts() # $ easy_install[%s]
   - distutils.setup_keywords()


- pip.commands._
  https://github.com/pypa/pip/tree/master/pip/commands

And then how does site.py interact with importlib and .pth files?
- https://github.com/python/cpython/blob/master/Lib/importlib/abc.py
-
https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py
  - source_from_cache()


On Saturday, November 26, 2016, Nick Coghlan  wrote:

> On 26 November 2016 at 19:34, Wes Turner  > wrote:
> > On Friday, November 25, 2016, Nick Coghlan  > wrote:
> >> By contrast, if we only propose deprecating "import" lines in ".pth"
> >> files, and also propose a more explicit approach to automatic code
> >> execution at interpreter startup, then it's only folks relying on the
> >> arbitrary-code-execution feature that would incur any migration costs.
> >> The language level pay-offs to justify that cost would be:
> >>
> >> - simplification of the current systems for implicit code execution at
> >> start-up
> >
> > There's
> >
> >  python -m site
> >
> > But is there a -m tool for checking .pth files?
> > (For code in .pth.py files)?
>
> Not currently, there's only the code that implicitly gets called from
> site.main() (which is in turn called as a side effect of importing the
> site module).
>
> So even though that machinery is already there in a pure Python module
> in every Python installation, none of it is exposed as a public
> Python-level API.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com|   Brisbane,
> Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 26 November 2016 at 19:34, Wes Turner  wrote:
> On Friday, November 25, 2016, Nick Coghlan  wrote:
>> By contrast, if we only propose deprecating "import" lines in ".pth"
>> files, and also propose a more explicit approach to automatic code
>> execution at interpreter startup, then it's only folks relying on the
>> arbitrary-code-execution feature that would incur any migration costs.
>> The language level pay-offs to justify that cost would be:
>>
>> - simplification of the current systems for implicit code execution at
>> start-up
>
> There's
>
>  python -m site
>
> But is there a -m tool for checking .pth files?
> (For code in .pth.py files)?

Not currently, there's only the code that implicitly gets called from
site.main() (which is in turn called as a side effect of importing the
site module).

So even though that machinery is already there in a pure Python module
in every Python installation, none of it is exposed as a public
Python-level API.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Wes Turner]

On Friday, November 25, 2016, Nick Coghlan  wrote:

> On 25 November 2016 at 12:26, Robert Collins  > wrote:
> > On 25 November 2016 at 14:04, Nick Coghlan  > wrote:
> >> The bad reputation of ".pth" doesn't generally stem from their normal
> >> usage (i.e. just listing absolute or relative directory names that the
> >> import system will then append to __path__).
> >>
> >> Rather, it stems from this little aspect: "Lines starting with import
> >> (followed by space or tab) are executed." (from
> >> https://docs.python.org/3/library/site.html )
> >
> > I think its also worth exploring a targeted, modern namespace aware
> replacement.
>
> Right. For a long time I thought "existing .pth files, only with the
> import line execution deprecated" was the only approach that stood any
> chance whatsoever of being adopted in a reasonable time frame, but we
> can technically use the "executable .pth file" trick ourselves to let
> people opt in to a new sys.path extension format on earlier Python
> versions.
>
> > That is - there are two, related, use cases for .pth files vis-a-vis
> > package installation. One is legacy namespace packages, which AFAICT
> > are still in use - the migration is "complicated". The second is
> > arguable a variant of that same thing: putting the current working dir
> > into the PYTHONPATH without putting it in PYTHONPATH.
>
> Third case: making zip archive contents available to applications
> without unpacking the archive first.
>
> > The former case may be sufficiently covered by (forget the pep #) that
> > we don't need to do anything,
>
> PEP 420, and I believe the only thing that can get tricky there now is
> figuring out how to do things in such a way that they work with both
> old-style namespace packages and the automatic Python 3 ones.
>
> > and the latter is certainly something
> > that we should be able to deliver without needing the turing complete
> > capability that you're suggesting. Something data driven rather than
> > code driven.
>
> While I agree with this, I think there's two pieces to it:
>
> - how to handle the data-driven use cases that we're entirely fine with
> - how to handle the arbitrary-code-execution use cases that we'd like
> to discourage, but don't want to make entirely impossible
>
> The problem with proposing an entirely new implicit path manipulation
> format is that if we deprecate ".pth" files entirely, then that hits
> *both* of those groups equally, while if we don't deprecate them at
> all, then neither group has an incentive to migrate to the new system.
>
> By contrast, if we only propose deprecating "import" lines in ".pth"
> files, and also propose a more explicit approach to automatic code
> execution at interpreter startup, then it's only folks relying on the
> arbitrary-code-execution feature that would incur any migration costs.
> The language level pay-offs to justify that cost would be:
>
> - simplification of the current systems for implicit code execution at
> start-up


There's

 python -m site

But is there a -m tool for checking .pth files?
(For code in .pth.py files)?

...

def cache_pth_sys_path(file='cached_pth.pyc'):
for (dirpath, pth, ouptut) in iter_pth_files():
with file(pth, 'r') as f:
print('\n# pth %r' % (dirpath, pth), file=f)
print(output, file=f) # ...


def iter_pth_files():
for dirpath in sys.path:
for pth in sorted(glob.glob(*.pth, dirpath)):
output = file(pth, 'r', utf8).read().close()
# imports = ast_imports(output) # PERF
yield (dirpath, pth, output))



> - making ".pth" files less dangerous as a format
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com|   Brisbane,
> Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 25 November 2016 at 12:26, Robert Collins  wrote:
> On 25 November 2016 at 14:04, Nick Coghlan  wrote:
>> The bad reputation of ".pth" doesn't generally stem from their normal
>> usage (i.e. just listing absolute or relative directory names that the
>> import system will then append to __path__).
>>
>> Rather, it stems from this little aspect: "Lines starting with import
>> (followed by space or tab) are executed." (from
>> https://docs.python.org/3/library/site.html )
>
> I think its also worth exploring a targeted, modern namespace aware 
> replacement.

Right. For a long time I thought "existing .pth files, only with the
import line execution deprecated" was the only approach that stood any
chance whatsoever of being adopted in a reasonable time frame, but we
can technically use the "executable .pth file" trick ourselves to let
people opt in to a new sys.path extension format on earlier Python
versions.

> That is - there are two, related, use cases for .pth files vis-a-vis
> package installation. One is legacy namespace packages, which AFAICT
> are still in use - the migration is "complicated". The second is
> arguable a variant of that same thing: putting the current working dir
> into the PYTHONPATH without putting it in PYTHONPATH.

Third case: making zip archive contents available to applications
without unpacking the archive first.

> The former case may be sufficiently covered by (forget the pep #) that
> we don't need to do anything,

PEP 420, and I believe the only thing that can get tricky there now is
figuring out how to do things in such a way that they work with both
old-style namespace packages and the automatic Python 3 ones.

> and the latter is certainly something
> that we should be able to deliver without needing the turing complete
> capability that you're suggesting. Something data driven rather than
> code driven.

While I agree with this, I think there's two pieces to it:

- how to handle the data-driven use cases that we're entirely fine with
- how to handle the arbitrary-code-execution use cases that we'd like
to discourage, but don't want to make entirely impossible

The problem with proposing an entirely new implicit path manipulation
format is that if we deprecate ".pth" files entirely, then that hits
*both* of those groups equally, while if we don't deprecate them at
all, then neither group has an incentive to migrate to the new system.

By contrast, if we only propose deprecating "import" lines in ".pth"
files, and also propose a more explicit approach to automatic code
execution at interpreter startup, then it's only folks relying on the
arbitrary-code-execution feature that would incur any migration costs.
The language level pay-offs to justify that cost would be:

- simplification of the current systems for implicit code execution at start-up
- making ".pth" files less dangerous as a format

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Wed, Nov 23, 2016, at 07:22 PM, Donald Stufft wrote:
> I think at a minimum we should get PEP 518 support into pip first. I
> don't think layering more things on top of a PEP that isn't yet
> implemented is a good approach to this. 

I went to make a start on this, but I got stuck on whether and how pip
should create temporary environments to install build dependencies and
build wheels. I've posted more details on the issue:

https://github.com/pypa/pip/issues/3691
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Ronny Pfannschmidt]

actually editable installs can be made uninstallable trivially

in gumby-elf i would create a fake wheel with files inside to facilitate
the path building for namespaces,
and a local version number (so pip would create my exe files and
uninstall clean)


-- Ronny
On 24.11.2016 01:23, Daniel Holth wrote:
>
> I wouldn't be afraid of editable installs. They are trivial and
> involve building the package in place and putting a .pth file where it
> will be noticed. Specify editable packages can't necessarily be
> uninstalled in a standard way and you are done.
>
> The bespoke build tool tells pip where the package root is (where
> .dist-info will be written), usually . or ./src, then pip does .pth.
>
>
> On Wed, Nov 23, 2016, 17:16 Chris Barker  > wrote:
>
> On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver
> mailto:tho...@kluyver.me.uk>> wrote:
>  
>
> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop')
> into a
> given prefix. I don't think that specifying a prefix is
> sufficient to
> cover '--user' installation, which uses a different install
> scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the
> prefix and
> do a user install.
> b. Leave it as is, and do not support editable user
> installation (which
> would make me sad, as I do editable user installs regularly)
>
>
> Please, please, let's figure SOMETHING our here - editable
> installs (or "develop" installs) are a critical tool. Frankly, I
> don't know how anyone can develop a package without them.
>
> Back in the day I struggle mightily with kludging sys.path, and,
> relative imports that never really worked right, and on and on --
> it SUCKED.
>
> Then I discovered setuptools develop mode -- yeah! IN fact, I
> don't think I'd ever use setuptools at all if I didn't need it to
> get develop mode!
>
> c. Decide that editable installs are too fiddly to
> standardise, and
> leave it to users to invoke a tool directly to do an editable
> install.
>
>
> Not sure what that means -- does that mean that you couldn't get
> an editable isntall with pip? but rather you would do:
>
> setup.py develop   if you had setuptools as your build system, and 
>
> some_other_command  if you had some other build tool?
>
> Not too bad, but why not have a standard way to invoke develop
> mode? If the tool can support it, why not have a way for pip to
> tell an arbitrary build system to "please install this package in
> develop mode"
>
> On the other hand:
>
> I've always thought we were moving toward proper separation of
> concerns, in which case, pip should be focused on resolving
> dependencies and finding and installing packages. 
>
> Should it even be possible to build and install a package from
> source with pip?
>
> But if it is, then it might as well support editable installs as well.
>
> The complication I see here is that a tool can't know how to
> install in editable mode unless it knows about the python
> environment it it running in -- which is easy for a tool built
> with python, but a problem for a tool written some other way.
>
> However, I see from PEP 517:
>
>
> The build backend object is expected to have attributes which
> provide some or all of the following hooks. The
> commonconfig_settings argument is described after the individual
> hooks:
>
> def get_build_requires(config_settings):
> ...
>
>
> So I guess we can count on a Python front end, at least, so 1(a)
> should be doable.
>
> In fact, is the user-install issue any different for editable
> installs than regular ones?
>
> -CHB
>
>
>
>
>
>
>
>  
>
>
>
>  
>
> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the
> table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose
> that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing
> quoting.
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> 
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
>
> -- 
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 5

Re: [Distutils] PEP 517: Build system API

[Robert Collins]

On 25 November 2016 at 14:04, Nick Coghlan  wrote:
..
>
> The bad reputation of ".pth" doesn't generally stem from their normal
> usage (i.e. just listing absolute or relative directory names that the
> import system will then append to __path__).
>
> Rather, it stems from this little aspect: "Lines starting with import
> (followed by space or tab) are executed." (from
> https://docs.python.org/3/library/site.html )

I think its also worth exploring a targeted, modern namespace aware replacement.

That is - there are two, related, use cases for .pth files vis-a-vis
package installation. One is legacy namespace packages, which AFAICT
are still in use - the migration is "complicated". The second is
arguable a variant of that same thing: putting the current working dir
into the PYTHONPATH without putting it in PYTHONPATH.

The former case may be sufficiently covered by (forget the pep #) that
we don't need to do anything, and the latter is certainly something
that we should be able to deliver without needing the turing complete
capability that you're suggesting. Something data driven rather than
code driven.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nick Coghlan]

On 25 November 2016 at 09:10, Paul Moore  wrote:
> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy? I recall it was possible to end up with a lot of clutter, but
> that was going back to the egg days, I'm not sure if it's like that
> any more. Why not just have a single pth file, maintained by the build
> tool, for all editable installs? Most users would then only have one
> or two pth files

The bad reputation of ".pth" doesn't generally stem from their normal
usage (i.e. just listing absolute or relative directory names that the
import system will then append to __path__).

Rather, it stems from this little aspect: "Lines starting with import
(followed by space or tab) are executed." (from
https://docs.python.org/3/library/site.html )

As such, rather than purely being a way of extending "__path__" with
additional directories, path files are *also* a way of
programmatically customising a particular Python installation, without
risking filename collisions the way sitecustomize and usercustomize
do.

Since setuptools is the first or only encounter most folks have with
".pth" files, and it makes use of such lines to execute arbitrary code
at interpreter startup, the targeted criticism of "the 'import line'
feature of path files and the specific ways that setuptools uses that
feature are hard to debug when they break" morphed over time into the
blanket criticism of "path files are bad".

The main problem with fixing that is that folks really do rely on the
auto-execution feature, sometimes even without knowing it (e.g.
because setuptools uses it implicitly).

Therefore, in order to deprecate and remove the magic import lines,
we'd need to do something like adding an "__autorun__" directory to
site-packages and user site-packages as a long term replacement for:

- sitecustomize.py
- usercustomize.py
- executable import lines in path files

Supporting such a subdirectory in older Python versions could then be
bootstrapped via an "autorun.pth" file that provided the same start-up
logic that 3.7+ did.

I suspect this would actually be an improvement from a security
awareness perspective as well, as it would make it much clearer that
enabling implicit code execution at Python interpreter start-up is
trivial if you have write access to the site-packages or user
site-packages directory.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Greg Ewing]

Thomas Kluyver wrote:

- There's a feature called NTFS Junction Points, which is supposed to be
like symlinks, but only for directories.


Things might have changed, but last time I played with
junction points they seemed rather dangerous, because
if you deleted one using the GUI it deleted the contents
of the directory it was linked to. And it appeared
indistinguishible from a normal folder in the GUI, so
you got no warning that was about to happen.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 21:51, Thomas Kluyver  wrote:
> On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote:
>> Just curious - how does flit handle Windows for this? Symlinks aren't
>> really an option there (you need elevation to create a symlink).
>> Paul
>
> It largely doesn't at present; it started out as a personal tool for me,
> and I mostly use Linux. I have a few ideas about how to deal with
> Windows, though:
>
> - There's a feature called NTFS Junction Points, which is supposed to be
> like symlinks, but only for directories. I believe these can be created
> by regular users without elevated permissions. It might be possible to
> use this for packages, albeit not for single file directories.
> - Create a script/tool to give a user the permission bit that allows
> creating symlinks, so users who know what they're doing can enable it.
> Maybe there's some reason it's admin-only by default, though? It's safe
> enough for normal users on Unix, but I don't know how different Windows
> is.

However you do it, it needs elevation, which is going to be a problem.
I don't think symlinks per se are any worse on Windows (apart from the
unfamiliarity aspect, which also applies to junction points, which may
mean tools don't handle them gracefully). But because you can't use
them without elevation/permission fiddling, they simply aren't an
end-user concept on Windows.

> - Run a daemon to watch package folders and reinstall on any changes,
> something like Nathaniel mentioned with 'pip watch'. I probably wouldn't
> do this as part of flit, but I'd be happy to see it as a separate tool.
> As Nathaniel pointed out, this can actually support more stuff than
> setuptools editable installs, or my symlinks, because it can re-run
> build steps.

That sounds like it could also have some issues making it work cleanly
on Windows (would it leave a console window open?).

Honestly, I don't see what's so bad about pth files. They are a
standard supported Python approach. Maybe setuptools' use of them is
messy? I recall it was possible to end up with a lot of clutter, but
that was going back to the egg days, I'm not sure if it's like that
any more. Why not just have a single pth file, maintained by the build
tool, for all editable installs? Most users would then only have one
or two pth files.

Anyway, this is OT, and I'm sure you'll come up with something
appropriate as and when the need arises. As I say, I was mostly just
curious.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

I've made PRs against PEP 517 for:

Underscore to dash in build-backend:
https://github.com/python/peps/pull/139

1a: Add a user parameter to the install_editable hook
https://github.com/python/peps/pull/140

OR:

1c: Get rid of the install_editable hook
https://github.com/python/peps/pull/141

I'm leaning towards 1c; it can always be standardised in a separate PEP,
and an optional hook added back.

Thomas

On Thu, Nov 24, 2016, at 09:06 AM, Paul Moore wrote:
> On 24 November 2016 at 00:32, Nathaniel Smith  wrote:
> > Also note that just like we decided to split the basic pyproject.toml
> > proposal (now PEP 518) from the build system interface proposal (now
> > PEP 517), it might (probably) makes sense to split the editable
> > install part of the build system interface proposal from the rest,
> > just in the interests of making incremental progress.
> 
> This would be my preference. I'm assuming (again, incremental
> progress) that "pip install -e" will still work as at present for
> projects that don't use the new infrastructure, so it's hardly an
> urgent need.
> 
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 08:07 PM, Paul Moore wrote:
> Just curious - how does flit handle Windows for this? Symlinks aren't
> really an option there (you need elevation to create a symlink).
> Paul

It largely doesn't at present; it started out as a personal tool for me,
and I mostly use Linux. I have a few ideas about how to deal with
Windows, though:

- There's a feature called NTFS Junction Points, which is supposed to be
like symlinks, but only for directories. I believe these can be created
by regular users without elevated permissions. It might be possible to
use this for packages, albeit not for single file directories.
- Create a script/tool to give a user the permission bit that allows
creating symlinks, so users who know what they're doing can enable it.
Maybe there's some reason it's admin-only by default, though? It's safe
enough for normal users on Unix, but I don't know how different Windows
is.
- Run a daemon to watch package folders and reinstall on any changes,
something like Nathaniel mentioned with 'pip watch'. I probably wouldn't
do this as part of flit, but I'd be happy to see it as a separate tool.
As Nathaniel pointed out, this can actually support more stuff than
setuptools editable installs, or my symlinks, because it can re-run
build steps.

Thomas

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 19:50, Thomas Kluyver  wrote:
> I hate the thing setuptools does with .pth files all over the place, so the
> equivalent operation in flit symlinks packages to site-packages.

Just curious - how does flit handle Windows for this? Symlinks aren't
really an option there (you need elevation to create a symlink).
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 07:50 PM, Thomas Kluyver wrote:

> I made a PR to flit to handle this case better in uninstallation.

I meant a PR to *pip*.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Thomas Kluyver]

On Thu, Nov 24, 2016, at 12:23 AM, Daniel Holth wrote:

> I wouldn't be afraid of editable installs. They are trivial and
> involve building the package in place and putting a .pth file where it
> will be noticed. Specify editable packages can't necessarily be
> uninstalled in a standard way and you are done.
> The bespoke build tool tells pip where the package root is (where .dist-
> info will be written), usually . or ./src, then pip does .pth.


The way it's specified at present is for pip to ask the build tool
(setuptools, flit, etc.) to do an editable install by whatever means.


I hate the thing setuptools does with .pth files all over the place, so
the equivalent operation in flit symlinks packages to site-packages. I
made a PR to flit to handle this case better in uninstallation.


I think Nathaniel's right: editable installs are going to be a source
of controversity and complexity. Let's drop that hook from the PEP for
now (ie. 1c) - it's still useful without it, and the design of the
build interface allows extra hooks to be specified and easily added in
the future.


Does anyone feel strongly that we should keep the editable install hook
in the current PEP? If not, I'll make a PR removing it (and switching
the underscore to dash, as that seems to be the consensus).


I'm also happy to look into implementing PEPs 518 and 517 in pip, though
it might be a week or two before I get time.


Thanks,

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 24 November 2016 at 00:32, Nathaniel Smith  wrote:
> Also note that just like we decided to split the basic pyproject.toml
> proposal (now PEP 518) from the build system interface proposal (now
> PEP 517), it might (probably) makes sense to split the editable
> install part of the build system interface proposal from the rest,
> just in the interests of making incremental progress.

This would be my preference. I'm assuming (again, incremental
progress) that "pip install -e" will still work as at present for
projects that don't use the new infrastructure, so it's hardly an
urgent need.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Nathaniel Smith]

On Wed, Nov 23, 2016 at 3:14 PM, Chris Barker  wrote:
> On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver 
> wrote:
>
>>
>> Questions:
>> 1. Editable installs. The PEP currenly specifies a hook to do an
>> editable install (like 'pip install -e' or 'setup.py develop') into a
>> given prefix. I don't think that specifying a prefix is sufficient to
>> cover '--user' installation, which uses a different install scheme,
>> especially on Windows and OSX framework builds. We can:
>> a. Add an extra parameter 'user' to the hook, to override the prefix and
>> do a user install.
>> b. Leave it as is, and do not support editable user installation (which
>> would make me sad, as I do editable user installs regularly)
>
>
> Please, please, let's figure SOMETHING our here - editable installs (or
> "develop" installs) are a critical tool. Frankly, I don't know how anyone
> can develop a package without them.

Would a 'pip watch' command that watched your source tree and
rebuilt/reinstalled it whenever you edited a file work for you? What
do you do for conda packages? Does conda have an editable install
equivalent?

The problem with editable installs is that they can't really be made
to work "properly", in the sense of having solid invariants and
fitting nicely into a rigorously defined metadata model -- they're
always going to have weird footguns around the metadata / .py files /
.so files getting out-of-sync with each other, and they make it harder
to improve pip's infrastructure around upgrading, because upgrading
requires pip to have a solid understanding of exactly what's
installed. So it's absolutely possible to have some useful hack, like
we do now, but useful hacks by their nature are hard to standardize:
standardization means "this is the final solution to this problem, it
won't need further tweaking", and editable installs really feel like
they might benefit from further tweaking. Or something.

Also note that just like we decided to split the basic pyproject.toml
proposal (now PEP 518) from the build system interface proposal (now
PEP 517), it might (probably) makes sense to split the editable
install part of the build system interface proposal from the rest,
just in the interests of making incremental progress.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Daniel Holth]

I wouldn't be afraid of editable installs. They are trivial and involve
building the package in place and putting a .pth file where it will be
noticed. Specify editable packages can't necessarily be uninstalled in a
standard way and you are done.

The bespoke build tool tells pip where the package root is (where
.dist-info will be written), usually . or ./src, then pip does .pth.

On Wed, Nov 23, 2016, 17:16 Chris Barker  wrote:

> On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver 
> wrote:
>
>
> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
>
>
> Please, please, let's figure SOMETHING our here - editable installs (or
> "develop" installs) are a critical tool. Frankly, I don't know how anyone
> can develop a package without them.
>
> Back in the day I struggle mightily with kludging sys.path, and, relative
> imports that never really worked right, and on and on -- it SUCKED.
>
> Then I discovered setuptools develop mode -- yeah! IN fact, I don't think
> I'd ever use setuptools at all if I didn't need it to get develop mode!
>
> c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.
>
>
> Not sure what that means -- does that mean that you couldn't get an
> editable isntall with pip? but rather you would do:
>
> setup.py develop   if you had setuptools as your build system, and
>
> some_other_command  if you had some other build tool?
>
> Not too bad, but why not have a standard way to invoke develop mode? If
> the tool can support it, why not have a way for pip to tell an arbitrary
> build system to "please install this package in develop mode"
>
> On the other hand:
>
> I've always thought we were moving toward proper separation of concerns,
> in which case, pip should be focused on resolving dependencies and finding
> and installing packages.
>
> Should it even be possible to build and install a package from source with
> pip?
>
> But if it is, then it might as well support editable installs as well.
>
> The complication I see here is that a tool can't know how to install in
> editable mode unless it knows about the python environment it it running in
> -- which is easy for a tool built with python, but a problem for a tool
> written some other way.
>
> However, I see from PEP 517:
>
>
> The build backend object is expected to have attributes which provide some
> or all of the following hooks. The commonconfig_settings argument is
> described after the individual hooks:
>
> def get_build_requires(config_settings):
> ...
>
>
> So I guess we can count on a Python front end, at least, so 1(a) should be
> doable.
>
> In fact, is the user-install issue any different for editable installs
> than regular ones?
>
> -CHB
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver 
wrote:


> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
>

Please, please, let's figure SOMETHING our here - editable installs (or
"develop" installs) are a critical tool. Frankly, I don't know how anyone
can develop a package without them.

Back in the day I struggle mightily with kludging sys.path, and, relative
imports that never really worked right, and on and on -- it SUCKED.

Then I discovered setuptools develop mode -- yeah! IN fact, I don't think
I'd ever use setuptools at all if I didn't need it to get develop mode!

c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.
>

Not sure what that means -- does that mean that you couldn't get an
editable isntall with pip? but rather you would do:

setup.py develop   if you had setuptools as your build system, and

some_other_command  if you had some other build tool?

Not too bad, but why not have a standard way to invoke develop mode? If the
tool can support it, why not have a way for pip to tell an arbitrary build
system to "please install this package in develop mode"

On the other hand:

I've always thought we were moving toward proper separation of concerns, in
which case, pip should be focused on resolving dependencies and finding and
installing packages.

Should it even be possible to build and install a package from source with
pip?

But if it is, then it might as well support editable installs as well.

The complication I see here is that a tool can't know how to install in
editable mode unless it knows about the python environment it it running in
-- which is easy for a tool built with python, but a problem for a tool
written some other way.

However, I see from PEP 517:


The build backend object is expected to have attributes which provide some
or all of the following hooks. The commonconfig_settings argument is
described after the individual hooks:

def get_build_requires(config_settings):
...


So I guess we can count on a Python front end, at least, so 1(a) should be
doable.

In fact, is the user-install issue any different for editable installs than
regular ones?

-CHB













> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Paul Moore]

On 23 November 2016 at 19:17, Brett Cannon  wrote:
> My vote is for 1c (easier to add 1a later), and dashes (for some reason I
> just like how they look more in config files).

I agree, 1c sounds like a reasonable starting place (but I don't tend
to use editable installs, so treat my views on this with caution :-))
Also, I like dashes. But I agree with Donald here. Layering multiple
unimplemented features sounds like a mistake. Can we not get PEP 518
implemented before we start on this again? Thomas, are you interested
in implementing the pip side of either or both of PEPs 517 and 518?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Matthew Brett]

Hi,

On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver  wrote:
> I'd like to push PEP 517 forwards again. This PEP specifies a general
> build system interface so that a source tree can specify a tool (such as
> flit), and pip can ask that tool to generate a wheel. This would allow
> us to install from an sdist or a VCS checkout without running a setup.py
> file.
>
> https://www.python.org/dev/peps/pep-0517/
>
> Developments since the last time this was discussed (to my knowledge):
> - It now uses the pyproject.toml file specified in PEP 518, which was
> already accepted. 517 originally specified a JSON file; 518 explains why
> TOML is preferred (basically: comments).
> - I have implemented the proposed build-system API in a pull request for
> Flit; this has been fairly straightforward so far.
> https://github.com/takluyver/flit/pull/77
>
> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
> c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.

I think the standard advice nowadays is to do user installs via pip,
so I would vote for some mechanism to do this - option a).

> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.

dash seems good to me too,

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Fred Drake]

On Wed, Nov 23, 2016 at 2:17 PM, Brett Cannon  wrote:
> My vote is for 1c (easier to add 1a later), and dashes (for some reason I
> just like how they look more in config files).

I'm glad I'm not the only one!  Configuration shouldn't require the
use of the hated 'shift' key.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Donald Stufft]

My 2 cents:

I think at a minimum we should get PEP 518 support into pip first. I don't 
think layering more things on top of a PEP that isn't yet implemented is a good 
approach to this. 

Sent from my iPhone

> On Nov 23, 2016, at 9:35 AM, Thomas Kluyver  wrote:
> 
> I'd like to push PEP 517 forwards again.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Brett Cannon]

Thanks for starting this up again!

My vote is for 1c (easier to add 1a later), and dashes (for some reason I
just like how they look more in config files).

On Wed, Nov 23, 2016, 06:36 Thomas Kluyver,  wrote:

> I'd like to push PEP 517 forwards again. This PEP specifies a general
> build system interface so that a source tree can specify a tool (such as
> flit), and pip can ask that tool to generate a wheel. This would allow
> us to install from an sdist or a VCS checkout without running a setup.py
> file.
>
> https://www.python.org/dev/peps/pep-0517/
>
> Developments since the last time this was discussed (to my knowledge):
> - It now uses the pyproject.toml file specified in PEP 518, which was
> already accepted. 517 originally specified a JSON file; 518 explains why
> TOML is preferred (basically: comments).
> - I have implemented the proposed build-system API in a pull request for
> Flit; this has been fairly straightforward so far.
> https://github.com/takluyver/flit/pull/77
>
> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
> c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.
>
> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig