Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nathaniel Smith]

On Fri, Jul 7, 2017 at 8:27 PM, Nick Coghlan  wrote:
> The latest round of discussions have been enlightening, as they have
> allowed us to articulate that from pip's point of view, the key
> requirement is to be able to tell a backend not to include anything
> that wouldn't be included when building via an sdist.
>
> From flit's point of view, Tomas wants frontends to able to express a
> preference between two different failure modes:
>
> 1. The frontend wants the wheel build to *guarantee* it exactly
> matches going via the sdist path
> 2. The frontend wants a working wheel build more than it cares about
> matching the sdist path

Both of these are handled neatly by my draft posted at the beginning
of this thread.

OTOH this whole 11th hour discussion of forcing every build system to
have in-tree and out-of-tree build support is solving some other
problem. I'm not entirely sure what that problem is -- I don't think
anyone has articulated it. Your "key requirement" is technically
vacuous -- by definition, *any* correct build backend will only
include the things that would be included when building via an sdist.

Some possible problems that I've seen mentioned in the thread include:

- pip doesn't trust build systems to properly support incremental
builds, so it wants to force it to throw away the build artifacts
after every build
- pip wants to enforce builds going via sdists
- Thomas / Ralf / I are frustrated at the idea of not supporting
incremental builds

But the in-tree/out-of-tree build proposal doesn't address any of
these problems. Part of the support for it seems to be that it
*sounds* like it might somehow provide a compromise between the folks
arguing over what pip should do by default, because it provides a way
of doing incremental builds that kind of looks like pip's current hack
for doing non-incremental builds. But I think this is illusory. We
still have to argue over what pip will actually do by default; all
this does is replace 1 way of doing incremental builds with 2 ways,
and if pip doesn't want to do incremental builds it'll ignore both.

Seriously, what problem is this solving? How can we even have a
discussion about whether it's the best solution when we don't know
that?

In my previous emails I was trying to avoid getting into the
nitty-gritty of actually critiquing the proposal, because I think it's
a fundamental mistake to even try to hash out this kind of complex
design at the last minute; for every issue I think of now there's
probably another one that none of us have noticed yet. But since
everyone else seems so gung-ho to ship this thing without even having
that minimal discussion, here are some concerns and questions:

If we require every project to support both in-tree and out-of-tree
builds, then projects that don't really support out-of-tree builds now
need to implement the copytree hack themselves, and they might get it
wrong. For example, if you don't correctly clear out the old build
tree before copying, your new build could potentially be corrupted by
artifacts from your old build. (I'm having flashbacks to the bug
reports we get on numpy from people who used setup.py install to
upgrade, and because it doesn't uninstall the old version they end up
with some combination of old and new versions overlaid on top of each
other.) This is a whole new potential failure mode that this proposal
is introducing. Is that acceptable?

In fact, I'm guessing that pip will not actually cache build
directories and re-use them, and will only ever supply empty
directories as the out-of-tree build dir. This means that lots of
projects are likely to be released without ever testing their
build-with-a-non-empty-out-of-tree-build-dir path, and thus it won't
work. What are the chances that this turns into another feature that
exists in theory but in practice it rusts over and can't be used,
because trying to do so would break too often?

What happens if a project switches from scons to cmake or vice-versa,
and they get passed a build tree that contains foreign build
artifacts? Are they prepared to detect this and do something sensible?
Traditionally reusing a build context (either an in-place build tree
or an out-of-place build tree) is something that only developers do,
and they do it manually, so this isn't a problem -- you just send an
occasional email to the list saying "hey I just flipped the switch on
the build systems, make sure to do a 'git clean' before your next
build", or people just reflexively do a 'git clean' any time they get
weird results. But now we're proposing to make this a first-class
feature to be used by automatic unsupervised build pipelines; are they
going to end up producing garbage?

Are frontends allowed to move the out-of-tree build directory to
another parent directory? Another filesystem? Another machine? Another
operating system? what do you they have to preserve if they try?
timestamps? inode numbers? (Using inode numbers as part of a change
detection algorithm

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Fri, Jul 14, 2017, at 11:18 AM, Nathaniel Smith wrote:
> OTOH this whole 11th hour discussion of forcing every build system to
> have in-tree and out-of-tree build support is solving some other
> problem. 

It is not my intention to force build systems to support either of
these.

- Where build systems only support out-of-tree builds, it should be
trivial to create a temporary directory when build_directory is not
passed.
- Where build systems only support in-tree builds, they should copy the
necessary files to build_directory and run an in-tree build there. This
is more complex, but it appears to be non-negotiable that there is some
way of building without affecting the source directory, so whatever the
interface is, we need some way to do this.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nathaniel Smith]

On Fri, Jul 14, 2017 at 3:32 AM, Thomas Kluyver  wrote:
> it appears to be non-negotiable that there is some
> way of building without affecting the source directory, so whatever the
> interface is, we need some way to do this.

But this is confusing the means with the ends. Obviously no cares
about that *per se*; they want it because it accomplishes something
they actually care about.

Maybe because they don't trust build tools to do incremental builds.
Maybe because they want to exercise the sdist path to reduce chances
for error. Maybe because pip has always done it that way. What is that
thing? What are the advantages of this design, as compared to the
fallback of doing unconditional copytree (like pip does now and may
well continue doing for years to come regardless of what we say here),
or the slightly fancier fallback that my draft supports of attempting
to build an sdist and if that fails doing a copytree instead? Is this
the simplest way to accomplish the real goal? All the goals that I can
think of seem to allow for solutions that have fewer complications and
unknowns...

...and if pip's goal is to go via sdist whenever possible while always
being careful never to modify the source tree, then why did we end up
with a design where sdist generation is the one case that *is*
encouraged to modify the source tree? This doesn't make any sense.

I'm sorry for dragging this out -- I know you just want to get
something finished and stop arguing about this. So do I :-). And your
efforts to keep pushing this stone up the hill are much appreciated. I
just think we should... find a shorter hill and declare victory over
that.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 14 July 2017 at 20:18, Nathaniel Smith  wrote:
> On Fri, Jul 7, 2017 at 8:27 PM, Nick Coghlan  wrote:
> Some possible problems that I've seen mentioned in the thread include:
>
> - pip doesn't trust build systems to properly support incremental
> builds, so it wants to force it to throw away the build artifacts
> after every build

It's less that, and more pip wanting to ensure that the default
publisher experience is reasonable close to the default end user
experience, in order to increase the likelihood that publishers ship
working sdists and wheel files, even if they haven't learned about the
full suite of available pre-release testing tools yet (or have chosen
not to use them).

> - pip wants to enforce builds going via sdists

This isn't a requirement, just an option we want to enable (and can't
avoid enabling).

> - Thomas / Ralf / I are frustrated at the idea of not supporting
> incremental builds

This is why we're allowing frontends to request in-place builds.

And that's been the main discovery of the last round of discussions -
we'd been talking past each other, and what we actually wanted was for
the backend interface to accurately model the in-place/out-of-tree
discussion that has been common to most build systems at least since
autotools (hence my catalog of them earlier in the thread).

This means we're no longer attempting to invent anything new, we're
just copying the extensive prior art around out-of-tree builds, and
will deal with the consequences.

Regarding your questions:

1. What if backends get their out-of-tree support wrong?

That's fine, it can be reported as a bug, fixed, and frontends and
redistributors can use pip constraint files to enforce minimum
versions even if publishers don't update their pyproject.toml files
(plus frontends will grab the latest version of build backends by
default).

2. What happens if re-using build directories isn't tested properly?

I'd be genuinely surprised if any frontend ever did this by default
(excluding in-place builds). Instead, it would be application
integrators doing it as part of something like BitBucket Pipelines or
OpenShift ImageStreams, and if it didn't work properly, they'd stop
caching that particular directory.

3. What happens if an upgrade breaks incremental builds?

The same thing that happens if an upgrade straight-up breaks: either
it gets reported as a bug and fixed, or else end users figure out a
workaround and move on.

4. Do we enforce not modifying the source directory?

No. We don't even enforce backends not doing "rm -rf ~". If folks want
anything like that enforced, then they still need to use ephemeral
build servers.

5. What happens if you try to switch from in-place to out-of-place in
one tree and it breaks?

Don't do that, then (this is a publisher-only issue that won't impact
end users).

6. Is native out-of-tree build support common in practice?

Yes, as they're a requirement for inclusion in Debian.

7. Why include in-place support in the API then?

Because some folks (including you) would like to include incremental
build support, and because if we don't support it explicitly backends
will still have to deal with the "wheel_directory == build_directory"
case.

8. But what if we make a design mistake and can't fix it?

Like all other recent PyPA specifications, PEP 517 will go through a
Provisional acceptance period where it is accepted for implementation
in the default toolset, but explicitly subject to revision based on
that real world feedback.

But even if something does still get through that second period of
review, the whole reason we opted for the Python backend API over the
CLI based one is that it's much easier to evolve (since we can do
getattr checks and so forth, as well as straight up declarations of
API version support)

9. Why not wait and then add a new backend capability requirement later?

Waiting to add the requirement won't provide us with any more data
than we already have, but may give backend implementors the impression
they don't need to care about out-of-tree build support. This is also
our first, last, and only chance to make out-of-tree build support a
*mandatory* backend requirement that frontends can rely on - if we add
it later, it will necessarily only be optional.

By contrast, if we include the out-of-tree build feature in the first
version, and it's straightforward for frontends to work with and
backends to implement, then cool, while if it proves problematic, then
we can review the design decision based on that additional
information.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 14 July 2017 at 11:32, Thomas Kluyver  wrote:
> On Fri, Jul 14, 2017, at 11:18 AM, Nathaniel Smith wrote:
>> OTOH this whole 11th hour discussion of forcing every build system to
>> have in-tree and out-of-tree build support is solving some other
>> problem.
>
> It is not my intention to force build systems to support either of
> these.
>
> - Where build systems only support out-of-tree builds, it should be
> trivial to create a temporary directory when build_directory is not
> passed.
> - Where build systems only support in-tree builds, they should copy the
> necessary files to build_directory and run an in-tree build there. This
> is more complex, but it appears to be non-negotiable that there is some
> way of building without affecting the source directory, so whatever the
> interface is, we need some way to do this.

It's "non-negotiable" in the sense that we (the pip developers) have
in the past had to deal with issues being reported that turned out to
be because a developer had unexpected files in their build directory
that resulted in the build being incorrect. That's not a case of "not
trusting build systems to do incremental builds", it's direct evidence
that developers sometimes make mistakes and think it's pip's fault
(until we explain their mistake to them).

The plan for pip, to address this ongoing issue, is to provide out of
tree builds, so that we can ensure that the developer is protected
against such mistakes. The UI for out of tree builds is not decided
finally yet (for example, whether it's the default, or if a flag is
needed). The reason for this is because current tools *make it
impossible* for us to implement this (with one proviso, it appears
that no-one ever actually tried the build a sdist, then build from the
sdist route - I thought we had and had hit issues caused by certain
tools, but I was apparently wrong).

So rather than back ourselves into a new corner where pip can't
provide out of tree builds to solve this issue, we need some means of
doing this in PEP 517. The option to do it via sdists, or via "copy
tree" hooks, became messy and was a problem for flit, so we took a
second look at it and agreed that we'd got hung up on not being
willing to trust backends to do out of tree builds. The new proposal
was to add a means for frontends to request an out of tree build, and
trust backends to provide that.

I'm happy with the current proposal. We shouldn't have a problem
trusting backends, and now we have a means of asking for what we want,
so everything is fine.

If the issue here is that front ends shouldn't be allowed to even
*want* to do out of tree builds, I'm not sure what I can further say
to convince you. We have actual bugs which we can solve with out of
tree builds, and we can't think of another way to solve them.
(Unfortunately, I can't find the issue in pip where we have discussed
implementing out of tree builds, so I can't give you a pointer to
actual bug reports - and anyway, most if not all of them will have
been worked around by the developer cleaning up his or her source
tree).

I hope this clarifies a little - I wasn't too keen on seeing the
requirement described as "non-negotiable" (or of it being linked to
incremental builds), when it's actually nothing other than an
identified need that addresses a real world problem frontends have to
deal with (of not-always-pristine build directories).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 14 July 2017 at 20:59, Nathaniel Smith  wrote:
> On Fri, Jul 14, 2017 at 3:32 AM, Thomas Kluyver  wrote:
>> it appears to be non-negotiable that there is some
>> way of building without affecting the source directory, so whatever the
>> interface is, we need some way to do this.
>
> But this is confusing the means with the ends. Obviously no cares
> about that *per se*; they want it because it accomplishes something
> they actually care about.
>
> Maybe because they don't trust build tools to do incremental builds.
> Maybe because they want to exercise the sdist path to reduce chances
> for error. Maybe because pip has always done it that way. What is that
> thing? What are the advantages of this design, as compared to the
> fallback of doing unconditional copytree (like pip does now and may
> well continue doing for years to come regardless of what we say here),
> or the slightly fancier fallback that my draft supports of attempting
> to build an sdist and if that fails doing a copytree instead? Is this
> the simplest way to accomplish the real goal? All the goals that I can
> think of seem to allow for solutions that have fewer complications and
> unknowns...
>
> ...and if pip's goal is to go via sdist whenever possible while always
> being careful never to modify the source tree, then why did we end up
> with a design where sdist generation is the one case that *is*
> encouraged to modify the source tree? This doesn't make any sense.

You're confusing two different aspects of the design here:

- the pip developer's desire for publisher flows to closely match end
user flows to reduce the risk of novice publishers shipping broken
archives
- the general build tool convention that out-of-tree builds shouldn't
modify the source tree

pip's design requirement is met by pip defaulting to doing build_sdist
and then doing build_wheel based on that sdist, so that's the simplest
possible design we could approve and ship.

The question then is what the fallback should be when build_sdist
fails, but only a wheel archive or installed package is needed.

Candidates:

- fail anyway (unecessarily poor UX if the wheel build would have worked)
- build in-place (reasonable option)
- build out-of-tree (reasonable option)
- shutil.copytree then build in-place (status quo for legacy sdists,
but problematic due to VCS directories & other large sets of data)

The design that PEP 517 has settled on is to say that since both
in-place and out-of-tree builds are reasonable things for a frontend
to request, the *API* will allow frontends to request either an
in-place build ("build_directory is None" or
"os.path.samefile(build_directory, os.getcwd())") or an out-of-tree
build ("os.path.samefile(build_directory, os.getcwd())").

We're not inventing the concept of an out-of-tree build, we're just
choosing to make it a requirement for PEP 517 backends to support
them. This is low risk, since by pushing out-of-tree build support to
the backend, we also make it straightforward for the *backend* to tell
publishers how to define a filtered shutli.copytree operation if the
backend doesn't natively support out-of-tree builds. That way we don't
need to standardise on a universal protocol for defining which files
are needed for the wheel building process - that can be a backend
dependent operation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 14 July 2017 at 21:23, Nick Coghlan  wrote:
> The design that PEP 517 has settled on is to say that since both
> in-place and out-of-tree builds are reasonable things for a frontend
> to request, the *API* will allow frontends to request either an
> in-place build ("build_directory is None" or
> "os.path.samefile(build_directory, os.getcwd())") or an out-of-tree
> build ("os.path.samefile(build_directory, os.getcwd())").

Oops, forgot to add the "not" after copying-and-pasting the samefile
call: out-of-tree builds are the inverse of out-of-tree builds, and
hence indicated by "build_directory is not None and not
os.path.samefile(build_directory, os.getcwd())".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP 420 on python2

[Asmodehn Shade]

Hi everyone,

I m very new around here, I just wanted to let you know that while writing
a custom importer for py2 and py3, I ended up backporting some python3
libimport code to python2.

The code is over there : https://github.com/asmodehn/filefinder2
This way I now support importing packages without __init__.py in python2.
It was simpler for my usecase than using pkg_resources or pkgutil.

So if anyone is interested in that, let me know, while I still have my head
around this.

Cheers !
--
AlexV
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 420 on python2

[Nick Coghlan]

On 14 July 2017 at 12:12, Asmodehn Shade  wrote:
> Hi everyone,
>
> I m very new around here, I just wanted to let you know that while writing a
> custom importer for py2 and py3, I ended up backporting some python3
> libimport code to python2.
>
> The code is over there : https://github.com/asmodehn/filefinder2
> This way I now support importing packages without __init__.py in python2.
> It was simpler for my usecase than using pkg_resources or pkgutil.

Nice!

While it's more intrusive (since it essentially replaces the entire
Python 2 import system with the Python 3 one), you may also find
https://pypi.python.org/pypi/importlib2 of interest.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Fri, Jul 14, 2017, at 11:59 AM, Nathaniel Smith wrote:
> What are the advantages of this design, as compared to the
> fallback of doing unconditional copytree (like pip does now and may
> well continue doing for years to come regardless of what we say here),

I don't think pip currently does copytree. Naively copying everything
may be very slow, especially on Windows, because it includes things like
the .git subdirectory. If you exclude the .git subdirectory, you break
tools like setuptools_scm which rely on the VCS for version numbers.

> or the slightly fancier fallback that my draft supports of attempting
> to build an sdist and if that fails doing a copytree instead?

Having a fallback which may be pathologically slow seems like bad UX
(it's hard to see what's going wrong), and I don't want whatever is
specified to hinge on generating sdists.

> ...and if pip's goal is to go via sdist whenever possible while always
> being careful never to modify the source tree, then why did we end up
> with a design where sdist generation is the one case that *is*
> encouraged to modify the source tree? This doesn't make any sense.

Sdist generation is *not* encouraged to modify the source tree. It is
encouraged to avoid generating significant artifacts, because the build
steps should be done after extracting the sdist. But if the backend
wants to generate intermediates during sdist creation, it is free to
make a temporary directory, or use an external cache mechanism.

> I'm sorry for dragging this out -- I know you just want to get
> something finished and stop arguing about this. So do I :-). And your
> efforts to keep pushing this stone up the hill are much appreciated. I
> just think we should... find a shorter hill and declare victory over
> that.

I appreciate the enthusiasm, but I feel like we're arguing about whether
that hill over there is smaller when we're almost at the top of this
one. ;-)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

The PEP process is awful. Sorry. Thank you for helping to make this happen.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Donald Stufft]

> On Jul 14, 2017, at 9:46 AM, Thomas Kluyver  wrote:
> 
> On Fri, Jul 14, 2017, at 11:59 AM, Nathaniel Smith wrote:
>> What are the advantages of this design, as compared to the
>> fallback of doing unconditional copytree (like pip does now and may
>> well continue doing for years to come regardless of what we say here),
> 
> I don't think pip currently does copytree. Naively copying everything
> may be very slow, especially on Windows, because it includes things like
> the .git subdirectory. If you exclude the .git subdirectory, you break
> tools like setuptools_scm which rely on the VCS for version numbers.


Still processing this new direction, but just FYI we *do* do copy tree today. 
It is naive and it is slow in many cases.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nathaniel Smith]

On Fri, Jul 14, 2017 at 4:00 AM, Nick Coghlan  wrote:
> On 14 July 2017 at 20:18, Nathaniel Smith  wrote:
>> On Fri, Jul 7, 2017 at 8:27 PM, Nick Coghlan  wrote:
>> Some possible problems that I've seen mentioned in the thread include:
>>
>> - pip doesn't trust build systems to properly support incremental
>> builds, so it wants to force it to throw away the build artifacts
>> after every build
>
> It's less that, and more pip wanting to ensure that the default
> publisher experience is reasonable close to the default end user
> experience, in order to increase the likelihood that publishers ship
> working sdists and wheel files, even if they haven't learned about the
> full suite of available pre-release testing tools yet (or have chosen
> not to use them).

This is exactly the opposite of what Paul says in his message posted
at about the same time as yours... AFAICT his argument is that build
artifact leakage is exactly what pip is worried about, and if anything
he's annoyed at me because he thinks I'm downplaying the problem :-).
(My impression is that Donald's position might be closer to what you
wrote, though.)

>> - pip wants to enforce builds going via sdists
>
> This isn't a requirement, just an option we want to enable (and can't
> avoid enabling).

Donald sure seemed to be insisting on it as a requirement earlier, at
least. And the *entire cause* of this whole discussion was conflict
between this requirement and flit not wanting to support building
sdists from unpacked sdists, so characterizing it as some automatic
thing that doesn't even need to be talked about seems really weird.

>> - Thomas / Ralf / I are frustrated at the idea of not supporting
>> incremental builds
>
> This is why we're allowing frontends to request in-place builds.

Wait, what? That's not what any of us asked for at all. Ralf and I
want incremental builds, which is orthogonal to the in-place versus
out-of-place distinction, and Thomas wants pip not to call sdist due
to some combination of inefficiency and lack-of-support-in-some-cases,
which is also orthogonal to the in-place versus out-of-place
distinction. (In retrospect I probably shouldn't have lumped Thomas /
Ralf / I together there, those really are different motivations,
though they lead to similar places.)

You're trying to shut me down by making these pronouncements about
what PEP 517 is going to be, and yet you don't seem to even understand
what requirements people are talking about. I'm sorry if that's rude,
and I'm not angry at you personally or anything, but I am frustrated
and I don't understand why I seem to have to keep throwing myself in
front of these ideas that seem so poorly motivated.

> And that's been the main discovery of the last round of discussions -
> we'd been talking past each other, and what we actually wanted was for
> the backend interface to accurately model the in-place/out-of-tree
> discussion that has been common to most build systems at least since
> autotools (hence my catalog of them earlier in the thread).

This statement seems just... completely wrong to me. I can't see any
way that the in-place/out-of-place distinction matches anything we
were talking about earlier. Have we been reading the same threads?
There was the whole thing about pip copying trees, of course, but
that's a completely different thing. The motivation there is that they
want to make sure that two builds don't accidentally influence each
other. If they don't trust the build system, then this requires
copying to a new tree (either via copytree or sdist). If they do trust
the build system, then there are other options -- you can do an
in-place build, or have a flag saying "please 'make clean' before this
build", or something. So out-of-place builds are either insufficient
or unnecessary for what pip wants. Copying the source tree and
out-of-place builds are completely different things with different use
cases; they happen to both involve external directories, but that's
mostly a false similarity I think.

And as for flit, it doesn't even have a distinction between in-place
and out-of-place builds...

> This means we're no longer attempting to invent anything new, we're
> just copying the extensive prior art around out-of-tree builds, and
> will deal with the consequences.

Forcing all build systems to support both in-place builds and
out-of-place builds adds substantial additional complexity, introduces
multiple new failure modes, and AFAICT is not very well suited to the
problems people have been worrying about.

> Regarding your questions:
>
> 1. What if backends get their out-of-tree support wrong?
>
> That's fine, it can be reported as a bug, fixed, and frontends and
> redistributors can use pip constraint files to enforce minimum
> versions even if publishers don't update their pyproject.toml files
> (plus frontends will grab the latest version of build backends by
> default).
>
> 2. What happens if re-using build directories isn't tested properly?
>
> I'd be 

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Donald Stufft]

> On Jul 14, 2017, at 2:24 PM, Nathaniel Smith  wrote:
> 
> On Fri, Jul 14, 2017 at 4:00 AM, Nick Coghlan  > wrote:
>> On 14 July 2017 at 20:18, Nathaniel Smith  wrote:
>>> On Fri, Jul 7, 2017 at 8:27 PM, Nick Coghlan  wrote:
>>> Some possible problems that I've seen mentioned in the thread include:
>>> 
>>> - pip doesn't trust build systems to properly support incremental
>>> builds, so it wants to force it to throw away the build artifacts
>>> after every build
>> 
>> It's less that, and more pip wanting to ensure that the default
>> publisher experience is reasonable close to the default end user
>> experience, in order to increase the likelihood that publishers ship
>> working sdists and wheel files, even if they haven't learned about the
>> full suite of available pre-release testing tools yet (or have chosen
>> not to use them).
> 
> This is exactly the opposite of what Paul says in his message posted
> at about the same time as yours... AFAICT his argument is that build
> artifact leakage is exactly what pip is worried about, and if anything
> he's annoyed at me because he thinks I'm downplaying the problem :-).
> (My impression is that Donald's position might be closer to what you
> wrote, though.)
> 
>>> - pip wants to enforce builds going via sdists
>> 
>> This isn't a requirement, just an option we want to enable (and can't
>> avoid enabling).
> 
> Donald sure seemed to be insisting on it as a requirement earlier, at
> least. And the *entire cause* of this whole discussion was conflict
> between this requirement and flit not wanting to support building
> sdists from unpacked sdists, so characterizing it as some automatic
> thing that doesn't even need to be talked about seems really weird.


Just to point out here that like many things, it’s complicated (tm). We have a 
number of issues that regularly come in on the pip side which really boil down 
to the build tool did something wrong and unexpected. Over time we’ve built up 
a number of strategies to attempt to work around those cases and as we look at 
a new system, we’re thinking of those cases while we evaluate it and going “Is 
this going to make those cases, better, worse, or not effect them?” and trying 
to push the design towards things that make those cases better rather than 
worse.

There is some differences I suspect in what specific problems me and Paul are 
each thinking of as well as well as what specific solutions to solving them we 
might like or even find acceptable. For better or worse pip doesn’t have a BDFL 
and for direction we generally follow rough consensus and working code. So 
“pip” doesn’t really have an official position on what we want from this, but 
Paul has a position and so do I.

To try and break down my thinking some more:

* I think it is a requirement that any PEP that proposes a mechanism for 
removing ``setup.py`` supports sdist and wheel building in it’s interface. 
Otherwise it sends a message that either one is an optional part of Python 
packaging, which is not a message we should be sending.

* I would *like* to remove a fairly common error case where we have a 
difference in the files that get included in a sdist and the files that get 
installed such that ``pip install .`` causes different files to be installed 
than ``python setup.py sdist && pip install dist/*.tar.gz``.

* I would *like* to remove another common error case where two subsequent 
builds end up affecting each other (at least by default) because of detritus 
left behind from the first build process. One example I can think of is if you 
have a library that can optionally link against some other library, you might 
``pip install .``, notice it didn’t link against that (because you didn’t have 
it installed) and then install it and ``pip install .`` again. Nothing would 
have changed in the directory to trigger a rebuild, so you’d get the behavior 
where it wouldn’t rebuild.

I have opinions about how the above things get solved, but I think only the 
first one is a hard and fast requirement where if the PEP doesn’t include it at 
all I simply cannot support the PEP at all. The other things are things I 
personally would very much prefer to solve, because it solves a common 
stumbling block for both new and experienced users alike.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 14 July 2017 at 19:24, Nathaniel Smith  wrote:
>>> - pip doesn't trust build systems to properly support incremental
>>> builds, so it wants to force it to throw away the build artifacts
>>> after every build
>>
>> It's less that, and more pip wanting to ensure that the default
>> publisher experience is reasonable close to the default end user
>> experience, in order to increase the likelihood that publishers ship
>> working sdists and wheel files, even if they haven't learned about the
>> full suite of available pre-release testing tools yet (or have chosen
>> not to use them).
>
> This is exactly the opposite of what Paul says in his message posted
> at about the same time as yours... AFAICT his argument is that build
> artifact leakage is exactly what pip is worried about, and if anything
> he's annoyed at me because he thinks I'm downplaying the problem :-).
> (My impression is that Donald's position might be closer to what you
> wrote, though.)

No, you've misunderstood my point completely. What I was trying to say
is *exactly* the same as what Nick said - we're just expressing it in
different terms.

I'm not sure how we can resolve this - from my perspective, everyone
else[1] is in agreement with the latest revision of the PEP. You have
some concerns, but I'm failing to understand what they are, except
that they seem to be based on a confusion about what "the rest of us"
want (I'm not trying to frame this as an "us against you" argument,
just trying to point out that you seem to be seeing conflicts in our
position that the rest of us don't). I'd really like to understand
your specific concerns, but we seem to be spending too much time
talking at cross purposes at the moment, and not making much progress.

[1] Donald's still thinking about it, but I'm fairly sure the proposal
aligns with what he's after, at least in broad terms.

So, maybe I can address some points I *think* you're making, and we
can try to understand each other that way.

1. Incremental builds. My understanding is that the current proposal
says *nothing* about incremental builds. There's nothing stopping
backends supporting them. On the other hand, pip at the moment doesn't
have a way for the user to request an incremental build, or request a
complete rebuild. As far as I know, there's nothing in pip's
documented behaviour that even suggests whether builds are incremental
(that's not to say we'll break what support there is arbitrarily, just
that it's not formalised). Solving the question of incremental builds
was, as far as I recall, agreed by everyone as something we'd defer to
a future PEP.

2. The need for out of place builds. Pip needs them - to solve a
certain class of issue that we've seen in real bug reports. You don't
seem convinced that we do need them, but I'm not sure what else I can
say here. You also seem concerned that if we get out of place builds,
that means that pip will use them to break functionality that you need
(specifically, incremental builds, as far as I can see). All I can say
to that is that you seem very unwilling to trust the pip developers to
take care to support our users' use cases. We're not going to
arbitrarily break users' workflows - and if you don't believe that I
can't convince you otherwise.

3. You are concerned that having to support in-place and out-of-place
builds is a significant burden for backends. Thomas has said it's OK
for flit (it's not a good fit, but he can support it). Daniel has
confirmed enscons can handle it. Nick has surveyed a number of other
build systems (that don't currently have backends, but could) and sees
no major problems. Can you give any concrete example of a case where a
realistic backend would find the requirement a burden? (Please don't
claim incremental builds as the reason - the PEP explicitly says that
backends can cache data from previous builds in the build_dir, which
allows incremental builds if the backend wants them).

The key point about *all* of the proposals we've had (build via sdist,
have a "prepare a build directory" hook, and the build directory
parameter) is to ensure that the wheels that are built when the
frontend requests them are the same as those that would be obtained if
the user were to build a sdist and then install from that. That's not
a case of "not trusting the backend", or "protecting against a bug in
the frontend or backend" - it's about making sure the *developer*
didn't make a mistake when modifying his project. That class of
mistake *does* happen - we've seen it reported by our end users.

To address some of your specific comments (apologies if I
misunderstand anything, it's just because I don't follow your basic
position at all, so things that seem obvious to you are confusing me
badly).

> You're trying to shut me down by making these pronouncements about
> what PEP 517 is going to be, and yet you don't seem to even understand
> what requirements people are talking about. I'm sorry if that's rude,
> and I'm not angry at you 

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 14 July 2017 at 21:23, Donald Stufft  wrote:
> * I think it is a requirement that any PEP that proposes a mechanism for
> removing ``setup.py`` supports sdist and wheel building in it’s interface.
> Otherwise it sends a message that either one is an optional part of Python
> packaging, which is not a message we should be sending.

I agree - although I accept that under certain conditions, certain
backends might fail to produce a sdist (for example flit can't produce
a sdist if the user doesn't have VCS tools installed). I'd rather they
didn't, but that's a quality of implementation issue with the backend,
and not something I think we should mandate.

> * I would *like* to remove a fairly common error case where we have a
> difference in the files that get included in a sdist and the files that get
> installed such that ``pip install .`` causes different files to be installed
> than ``python setup.py sdist && pip install dist/*.tar.gz``.

I agree - for me, this is an important improvement that we can make to
pip, and if the PEP makes it impossible for us to do this I'd be very
reluctant to accept it. We could fall back to building via sdist, but
if sdist creation can fail, that's a problem - so I'd have to rethink
my willingness to accept that build_sdist could fail if we didn't get
support for this.

> * I would *like* to remove another common error case where two subsequent
> builds end up affecting each other (at least by default) because of detritus
> left behind from the first build process. One example I can think of is if
> you have a library that can optionally link against some other library, you
> might ``pip install .``, notice it didn’t link against that (because you
> didn’t have it installed) and then install it and ``pip install .`` again.
> Nothing would have changed in the directory to trigger a rebuild, so you’d
> get the behavior where it wouldn’t rebuild.

I have no real opinion on this one. Or at least, I'm not willing to
get into fights about incremental build support at this time, so I'll
abstain here.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Daniel Holth]

I proposed the build directory parameter because the copytree hook made no
sense to me. It is not a perfect substitute but perhaps a configurable
build directory is nice on its own without having to satisfy all older
arguments in favor of copytree. I think true in-place builds are the
oddball (for example 2to3 or any build where sources have the same name as
outputs needs a build directory to put the translated .py files, otherwise
it would overwrite the source). What people think of as in-place builds in
distutils are usually just builds into the default build directory.

IIRC the main reason we are worried sdist generation would fail is when you
are trying to do new development from an unpacked sdist. Suggest not
worrying that this use case is a perfect experience. It's good enough for
quick patching, but if you need to do a new source release then you
probably have time to meet the requirements of your build backend.

Surely the pluggable build systems will compete on reliability, rather than
repeatedly making MANIFEST.in errors you will be able to just switch
build-backend.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Ralf Gommers]

On Sat, Jul 15, 2017 at 9:31 AM, Daniel Holth  wrote:

> I proposed the build directory parameter because the copytree hook made no
> sense to me. It is not a perfect substitute but perhaps a configurable
> build directory is nice on its own without having to satisfy all older
> arguments in favor of copytree. I think true in-place builds are the
> oddball (for example 2to3 or any build where sources have the same name as
> outputs needs a build directory to put the translated .py files, otherwise
> it would overwrite the source). What people think of as in-place builds in
> distutils are usually just builds into the default build directory.
>

That's not the interesting part, it doesn't matter if a build is done in
build/lib*/etc inside the repo or outside, what matters is that the final
build artifacts are placed back in the source tree. So a C extension will
have .c files in the tree, and after an inplace build it will have .c and
.so (but no .o !).

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 420 on python2

[Asmodehn Shade]

Hi,

I had a look at libimport2, but I found out it was a bit buggy and I
couldnt extend it as I had hoped.

The custom importer I m working on basically extends PathFinder, FileFinder
and SourceFileLoader, and I wanted to have the same implementation for py2
and py3, so I ended up reimplementing the parts of importlib I needed in
order to be able to extend from them, from py2 or py3, without any version
specific code.

Now I m trying to find the best way to test it to make sure py2 +
filefinder2 behaves like py3 as much as possible, so any insight would be
helpful :-)

Thanks !

On Jul 14, 2017 9:19 PM, "Nick Coghlan"  wrote:

> On 14 July 2017 at 12:12, Asmodehn Shade  wrote:
> > Hi everyone,
> >
> > I m very new around here, I just wanted to let you know that while
> writing a
> > custom importer for py2 and py3, I ended up backporting some python3
> > libimport code to python2.
> >
> > The code is over there : https://github.com/asmodehn/filefinder2
> > This way I now support importing packages without __init__.py in python2.
> > It was simpler for my usecase than using pkg_resources or pkgutil.
>
> Nice!
>
> While it's more intrusive (since it essentially replaces the entire
> Python 2 import system with the Python 3 one), you may also find
> https://pypi.python.org/pypi/importlib2 of interest.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Including soabi tag in --user path?

[Nathaniel Smith]

Hi all,

A user on the pywavelets mailing list seems to have gotten themselves
into a mess: AFAICT they did 'pip install --user pywavelets' using a
ucs4 python, and then started a ucs2 python and tried to import it,
which of course failed because these two python builds have
incompatible ABIs, even though they share the same --user directory.

In retrospect, this seems like an obvious oversight: right now the
--user path (on Linux) is something like

  ~/.local/lib/python3.5/site-packages

Wouldn't it make more sense for it to be something like

  ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages

so that Pythons with incompatible ABIs can't see each other's
packages? (That's sysconfig.get_config_var("SOABI").)

Obviously this can't really be fixed for old versions, but is this a
change that should be made in 3.7?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Ned Deily]

On Jul 15, 2017, at 00:22, Nathaniel Smith  wrote:
> A user on the pywavelets mailing list seems to have gotten themselves
> into a mess: AFAICT they did 'pip install --user pywavelets' using a
> ucs4 python, and then started a ucs2 python and tried to import it,
> which of course failed because these two python builds have
> incompatible ABIs, even though they share the same --user directory.
> 
> In retrospect, this seems like an obvious oversight: right now the
> --user path (on Linux) is something like
> 
>  ~/.local/lib/python3.5/site-packages
> 
> Wouldn't it make more sense for it to be something like
> 
>  ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
> 
> so that Pythons with incompatible ABIs can't see each other's
> packages? (That's sysconfig.get_config_var("SOABI").)
> 
> Obviously this can't really be fixed for old versions, but is this a
> change that should be made in 3.7?

I'm confused.  This shouldn't be an issue for Python 3.*.  First, there is no 
ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string 
representation. But even before that, PEP 3149 (in 3.2) is supposed to ensure 
that extension modules have unique names based on ABI, for example:

_psutil_osx.cpython-35m-darwin.so

This *is* a problem in Python 2.7 but that's not going to change now.

--
  Ned Deily
  n...@python.org -- []

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Nathaniel Smith]

On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily  wrote:
> On Jul 15, 2017, at 00:22, Nathaniel Smith  wrote:
>> A user on the pywavelets mailing list seems to have gotten themselves
>> into a mess: AFAICT they did 'pip install --user pywavelets' using a
>> ucs4 python, and then started a ucs2 python and tried to import it,
>> which of course failed because these two python builds have
>> incompatible ABIs, even though they share the same --user directory.
>>
>> In retrospect, this seems like an obvious oversight: right now the
>> --user path (on Linux) is something like
>>
>>  ~/.local/lib/python3.5/site-packages
>>
>> Wouldn't it make more sense for it to be something like
>>
>>  ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
>>
>> so that Pythons with incompatible ABIs can't see each other's
>> packages? (That's sysconfig.get_config_var("SOABI").)
>>
>> Obviously this can't really be fixed for old versions, but is this a
>> change that should be made in 3.7?
>
> I'm confused.  This shouldn't be an issue for Python 3.*.  First, there is no 
> ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string 
> representation.

Right, but there are still SOABI variations, e.g. debug builds versus
regular builds.

> But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension 
> modules have unique names based on ABI, for example:
>
> _psutil_osx.cpython-35m-darwin.so

Right, but this only helps if you install multiple builds of the same
package on top of each other. Which Debian is able to arrange, so PEP
3149 solves their problem. But the --user dir is managed by pip, and
if you try to install a cpython-35dm package on top of a cpython-35m
package, then pip will helpfully remove the first before installing
the second. (And this is really the only reasonable thing for pip to
do.) So in practice any given package in the --user dir can only
support one SOABI variant, even though all the SOABI variants use the
same dir.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Ned Deily]

On Jul 15, 2017, at 00:41, Nathaniel Smith  wrote:
> 
> On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily  wrote:
>> On Jul 15, 2017, at 00:22, Nathaniel Smith  wrote:
>>> A user on the pywavelets mailing list seems to have gotten themselves
>>> into a mess: AFAICT they did 'pip install --user pywavelets' using a
>>> ucs4 python, and then started a ucs2 python and tried to import it,
>>> which of course failed because these two python builds have
>>> incompatible ABIs, even though they share the same --user directory.
>>> 
>>> In retrospect, this seems like an obvious oversight: right now the
>>> --user path (on Linux) is something like
>>> 
>>> ~/.local/lib/python3.5/site-packages
>>> 
>>> Wouldn't it make more sense for it to be something like
>>> 
>>> ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
>>> 
>>> so that Pythons with incompatible ABIs can't see each other's
>>> packages? (That's sysconfig.get_config_var("SOABI").)
>>> 
>>> Obviously this can't really be fixed for old versions, but is this a
>>> change that should be made in 3.7?
>> 
>> I'm confused.  This shouldn't be an issue for Python 3.*.  First, there is 
>> no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string 
>> representation.
> 
> Right, but there are still SOABI variations, e.g. debug builds versus
> regular builds.

And those are covered in the file names, e.g. a debug build extension module 
would be:

_psutil_osx.cpython-35dm-darwin.so

> But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension 
> modules have unique names based on ABI, for example:
>> 
>> _psutil_osx.cpython-35m-darwin.so
> 
> Right, but this only helps if you install multiple builds of the same
> package on top of each other. Which Debian is able to arrange, so PEP
> 3149 solves their problem. But the --user dir is managed by pip, and
> if you try to install a cpython-35dm package on top of a cpython-35m
> package, then pip will helpfully remove the first before installing
> the second. (And this is really the only reasonable thing for pip to
> do.) So in practice any given package in the --user dir can only
> support one SOABI variant, even though all the SOABI variants use the
> same dir.

Well, that sounds like either a pip problem or a packager's problem.  The point 
of PEP 3149 was exactly to allow multiple builds of extension modules built 
with different ABIs to co-exist in the same directory.  You would have the same 
behavior with packages installed to the system site-packergs dir, right?  So 
what's special about --user?

--
  Ned Deily
  n...@python.org -- []

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 15 July 2017 at 07:31, Daniel Holth  wrote:
> I proposed the build directory parameter because the copytree hook made no
> sense to me. It is not a perfect substitute but perhaps a configurable build
> directory is nice on its own without having to satisfy all older arguments
> in favor of copytree. I think true in-place builds are the oddball (for
> example 2to3 or any build where sources have the same name as outputs needs
> a build directory to put the translated .py files, otherwise it would
> overwrite the source). What people think of as in-place builds in distutils
> are usually just builds into the default build directory.
>
> IIRC the main reason we are worried sdist generation would fail is when you
> are trying to do new development from an unpacked sdist. Suggest not
> worrying that this use case is a perfect experience. It's good enough for
> quick patching, but if you need to do a new source release then you probably
> have time to meet the requirements of your build backend.

This reminds me of something I should have mentioned on list, but made
the mistake of assuming would be equally clear to everyone else: the
last time that Thomas elaborated on his concerns about VCS metadata
potentially being unavailable, I realised there's a *very* common
build use case where we can safely assume that metadata will be
missing.

Specifically: source builds in systems like RPM, deb, and conda. Those
start with a release tarball, *not* a VCS checkout, and while they'll
likely start with the sdist rather than a raw tarball for most Python
projects, that *won't* be the case for projects that include generated
artifacts like Cython output files in their sdists.

So to handle that kind of scenario under PEP 517, we're going to
*need* wheel builds to work even when sdist builds would fail due to a
lack of VCS metadata.

That doesn't specifically say anything about in-tree vs out-of-tree
build support (aside from the fact that Debian's packaging policies
call for out-of-tree builds), but it does mean that having the backend
API *only* support the "source tree -> sdist -> wheel" build path is
genuinely insufficient for our full array of use cases.

However, backends that *don't* have any special requirements for
creating sdists will still be free to implement their out-of-tree
build support that way, which is what I'm proposing we do for the
example backend in the PEP:
https://github.com/python/peps/pull/310/files

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 420 on python2

[Nick Coghlan]

On 15 July 2017 at 14:07, Asmodehn Shade  wrote:
> Hi,
>
> I had a look at libimport2, but I found out it was a bit buggy and I couldnt
> extend it as I had hoped.

That's unfortunately entirely plausible, as the downside of importlib
co-evolving with the import implementation in the interpreter is that
some of its internal assumptions don't necessarily hold when it gets
backported to earlier versions :(

> The custom importer I m working on basically extends PathFinder, FileFinder
> and SourceFileLoader, and I wanted to have the same implementation for py2
> and py3, so I ended up reimplementing the parts of importlib I needed in
> order to be able to extend from them, from py2 or py3, without any version
> specific code.

Ah, very cool!

> Now I m trying to find the best way to test it to make sure py2 +
> filefinder2 behaves like py3 as much as possible, so any insight would be
> helpful :-)

If you aren't already, it may be worthwhile running some of your test
cases under "python -v" and seeing if the output is stable enough
across different Python versions that filtering stderr for lines
starting with "import pkg_under_test" gets you a suitable test case
that isn't overly fragile.

Aside from that, the main suggestion I can think of would be to look
for relevant test cases in the Python 3 import and importlib test
suites and see if you can adapt them for filefinder2.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Nathaniel Smith]

On Fri, Jul 14, 2017 at 9:48 PM, Ned Deily  wrote:
> On Jul 15, 2017, at 00:41, Nathaniel Smith  wrote:
>>
>> On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily  wrote:
>>> On Jul 15, 2017, at 00:22, Nathaniel Smith  wrote:
 A user on the pywavelets mailing list seems to have gotten themselves
 into a mess: AFAICT they did 'pip install --user pywavelets' using a
 ucs4 python, and then started a ucs2 python and tried to import it,
 which of course failed because these two python builds have
 incompatible ABIs, even though they share the same --user directory.

 In retrospect, this seems like an obvious oversight: right now the
 --user path (on Linux) is something like

 ~/.local/lib/python3.5/site-packages

 Wouldn't it make more sense for it to be something like

 ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages

 so that Pythons with incompatible ABIs can't see each other's
 packages? (That's sysconfig.get_config_var("SOABI").)

 Obviously this can't really be fixed for old versions, but is this a
 change that should be made in 3.7?
>>>
>>> I'm confused.  This shouldn't be an issue for Python 3.*.  First, there is 
>>> no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string 
>>> representation.
>>
>> Right, but there are still SOABI variations, e.g. debug builds versus
>> regular builds.
>
> And those are covered in the file names, e.g. a debug build extension module 
> would be:
>
> _psutil_osx.cpython-35dm-darwin.so
>
>> But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension 
>> modules have unique names based on ABI, for example:
>>>
>>> _psutil_osx.cpython-35m-darwin.so
>>
>> Right, but this only helps if you install multiple builds of the same
>> package on top of each other. Which Debian is able to arrange, so PEP
>> 3149 solves their problem. But the --user dir is managed by pip, and
>> if you try to install a cpython-35dm package on top of a cpython-35m
>> package, then pip will helpfully remove the first before installing
>> the second. (And this is really the only reasonable thing for pip to
>> do.) So in practice any given package in the --user dir can only
>> support one SOABI variant, even though all the SOABI variants use the
>> same dir.
>
> Well, that sounds like either a pip problem or a packager's problem.  The 
> point of PEP 3149 was exactly to allow multiple builds of extension modules 
> built with different ABIs to co-exist in the same directory.  You would have 
> the same behavior with packages installed to the system site-packergs dir, 
> right?  So what's special about --user?

For system-managed packages, we tend to get away with it because
Debian [1] is able to enforce that there's one .deb file that ships
all of the pure python files and all of the SOABI variants for
extension modules together, so everything remains consistent.

For pip managed packages this doesn't really make sense. You have to
somehow ensure that two different pip runs, using different versions
of python, are both allowed to install two copies of the same package
on top of each other into the same directory, and that the only
difference between them is the extension modules. And what do you do
if one SOABI Python install package X version Y, but then a different
SOABI Python tries to upgrade to package X version (Y+1)?

Another possible option would be to have both an SOABI-tagged
directory and a non-tagged directory on sys.path, and install packages
that contain extensions into one and pure-python packages into the
other. Allegedly this is what the purelib/platlib distinction in wheel
metadata is supposed to enable, I think, but in practice there are
definitely wheels out there that break this (e.g. tensorflow is
allegedly a purelib wheel), so I don't know if it works in practice.

And yeah, it might well make sense to do this for system site packages
too, but then there might be additional complications with Debian etc.
so I figured I'd start with the obvious case :-).

-n

[1] or whoever is distributing system packages, but PEP 3149 came from Debian.

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Nick Coghlan]

On 15 July 2017 at 14:41, Nathaniel Smith  wrote:
>> _psutil_osx.cpython-35m-darwin.so
>
> Right, but this only helps if you install multiple builds of the same
> package on top of each other. Which Debian is able to arrange, so PEP
> 3149 solves their problem. But the --user dir is managed by pip, and
> if you try to install a cpython-35dm package on top of a cpython-35m
> package, then pip will helpfully remove the first before installing
> the second. (And this is really the only reasonable thing for pip to
> do.) So in practice any given package in the --user dir can only
> support one SOABI variant, even though all the SOABI variants use the
> same dir.

Well, sort of - it's not intrinsic to *pip* or *Python* that this
happens, it's an artifact of the fact that even though the
compatibility tagging scheme allows for it, folks don't tend to
publish "multi-SOABI" wheel files (where the 35dm and 35m binaries are
in the same wheel), so installing one will uninstall the other.

Rather than attempting to change how site-packages works (which
wouldn't help until 3.7+ anyway), the main missing piece in the status
quo is the equivalent of a "mergewheel" operation that creates those
multi-SOABI files and thus allows both sets of binaries to be added to
the same environment (the same way Linux distros do for the system
site-packages directory).

Cheers,
Nick.

P.S. Folks on Python 2.7 are pretty much stuck, though - while
importlib2 and/or filefinder2 *might* be able to help, several of the
pieces needed to fix this properly are Python 3 only.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Ned Deily]

On Jul 15, 2017, at 01:51, Nick Coghlan  wrote:
> Well, sort of - it's not intrinsic to *pip* or *Python* that this
> happens, it's an artifact of the fact that even though the
> compatibility tagging scheme allows for it, folks don't tend to
> publish "multi-SOABI" wheel files (where the 35dm and 35m binaries are
> in the same wheel), so installing one will uninstall the other.

Right, which is why I asserted it's a packager's problem.

> Rather than attempting to change how site-packages works (which
> wouldn't help until 3.7+ anyway), the main missing piece in the status
> quo is the equivalent of a "mergewheel" operation that creates those
> multi-SOABI files and thus allows both sets of binaries to be added to
> the same environment (the same way Linux distros do for the system
> site-packages directory).

Without thinking too much about it, perhaps another option would be for 
packagers to provide two wheels: one containing all files needed for the 
non-debug case and a second wheel that only contains the debug extension 
modules (with their usual unique fila names) and that depends on the first 
wheel.

In any case, I agree that this isn't something that has come up often in the 
past but perhaps some best practices should be documented in the packaging user 
guide.  I added a comment to that effect in open issue #284:

https://github.com/pypa/python-packaging-user-guide/issues/284

--
  Ned Deily
  n...@python.org -- []

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Including soabi tag in --user path?

[Nick Coghlan]

On 15 July 2017 at 16:00, Ned Deily  wrote:
> On Jul 15, 2017, at 01:51, Nick Coghlan  wrote:
>> Rather than attempting to change how site-packages works (which
>> wouldn't help until 3.7+ anyway), the main missing piece in the status
>> quo is the equivalent of a "mergewheel" operation that creates those
>> multi-SOABI files and thus allows both sets of binaries to be added to
>> the same environment (the same way Linux distros do for the system
>> site-packages directory).
>
> Without thinking too much about it, perhaps another option would be for 
> packagers to provide two wheels: one containing all files needed for the 
> non-debug case and a second wheel that only contains the debug extension 
> modules (with their usual unique fila names) and that depends on the first 
> wheel.

This is essentially what Linux distros do - the debug packages are
distinct from the regular ones, but they also have different names, so
there's nothing keeping you from having both installed on a given
system at the same time.

However, there's a missing primitive for that in our case: we expect
wheel archives to define independent package trees, with namespaces
package being the only means of injecting your own files into another
wheel's packages, so the debug and non-debug builds conflict by
default.

This means that for the specific problem of injecting debug versions
of packages into a project, we're probably better off encouraging the
use of package shadowing idioms that are more akin to the way
LD_LIBRARY_PATH works than we are changing the way user site-packages
works in general. Conveniently, we also already offer the key building
block for that: virtual environments with --system-site-packages
enabled.

That is, if folks want to run the debug versions of their Python
extensions without having to duplicate all their pure Python modules,
we'd encourage them to:

  1. Create a virtualenv with --system-site-packages and SOABI=cp35dm
(for example)
  2. Install the debug versions of any components with extension
modules into that virtual environment (since SOABI=cp35m modules won't
load)
  3. Run the debug version from that venv

One of the key benefits of encouraging that model is that it's
general: it works for *any* case where you want to override the
default version of a package, not just those where you want to run it
against a different SOABI definition.

We're missing a useful building block for *sharing* debug binaries
between different venvs, but that's also a problem that's amenable to
more general solutions (with the main one that comes to mind being a
way to "chain" virtual environments, such that you can easily add a
*.pth file to a venv that injects the site-packages for one of more
other venvs into sys.path)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig