Re: [Distutils] A possible refactor/streamlining of PEP 517

[Donald Stufft]

> On Jul 5, 2017, at 2:02 AM, Nick Coghlan  wrote:
> 
> On 5 July 2017 at 15:49, Donald Stufft  wrote:
>> I’ve had a niggling feeling about this hook from the beginning, but I
>> couldn’t quite put my finger on it until Nathaniel’s email made me realize
>> it. I feel like this hook is really *only* useful for flit, and for other
>> projects it is largely either going to be completely redundant or be an
>> attractive nuisance that ends up only causing issues. It’s a pretty narrow
>> use case where this hook is both able to do something AND doesn’t have the
>> exact same requirements as build_sdist.
>> 
>> When I felt this was a more generic hook, I was OK with it, but I don’t
>> think it’s a good idea now that I’ve thought on it more and it feels
>> entirely ungeneric.
> 
> I don't think Thomas's plans for it are unusual, as it's normal for a
> build system to only be aware of the input files that are actually
> referenced by the build recipe, and also normal for published source
> archives to include additional files that *aren't* used by the build
> process for the binary artifacts.

Except in this case the build system and the thing that builds the binary 
artifacts are one in the same so it needs to know both of those things. This 
hook is only useful if you have two different mechanisms for declaring those 
types of files *AND* the requirements for the additional files imposes some 
other non-python level prerequisites on the system that could maybe be avoided.

Quite literally, the only case I can think of that fits into this is flit’s “I 
will use git to figure out additional files, but you will have to configure in 
a static file the name of the Python package (as in import package) that you’re 
distributing and I’ll just glom down that entire package directory”. I know 
setuptools/distutils doesn’t work this way nor do any of the plugins that I am 
aware of. Best I can tell numpy.distutils nor enscons.

Looking at https://github.com/takluyver/flit/blob/master/flit/sdist.py 
 (added in 
https://github.com/takluyver/flit/pull/106 
) it appears that flit *also* 
doesn’t work this way, when it builds the sdist today it is looking at the VCS 
tracking and including any file mentioned in the VCS directory. So unless flit 
changes it’s logic so it only uses the “is this file in a VCS” for non 
installed files, it also can’t use this hook without the VCS tools installed 
and guarantee that the output is the same.

So not only is this a hook that I think is only going to be used by flit, but 
flit itself can’t even use it right now without changing the logic on how they 
generate sdists to no longer reference the VCS for what installable files get 
added. The more I dig into this, the more I think Nathaniel is correct and 
we’re trying to add a hook without any real world experience guiding it’s 
inclusion that doesn’t actually solve the problem it’s trying to solve and 
which it’s primary use case is creating foot guns.

> 
> If you'd prefer some external validation for the concept, I see the
> "prepare_input_for_build_wheel" hook as fairly analagous to the
> "%prep" phase in the process of building an RPM:
> https://fedoraproject.org/wiki/How_to_create_an_RPM_package#.25prep_section
> 
> The current difference is that we expect backends to be able to cope
> with frontends *not* calling that implicitly when building from an
> unpacked sdist.
> 

I don’t think this is external validation at all and the only similarities I 
see between he two hooks is they both reference the concept of “preparing” in 
their names. The %prep’s hook in RPMs primary use case is running tar to unpack 
a tarball and running patch on the resulting unpacked tarball (in fact, this is 
so common they have a %autosetup which basically does that). It’s really an 
entirely different hook from what we’re discussing here.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Donald Stufft]

> On Jul 5, 2017, at 11:19 AM, Donald Stufft  wrote:
> 
> The more I dig into this, the more I think Nathaniel is correct and we’re 
> trying to add a hook without any real world experience guiding it’s inclusion 
> that doesn’t actually solve the problem it’s trying to solve and which it’s 
> primary use case is creating foot guns.


Thinking about this more, I think the right thing to do here is remove this 
from the standard for now, and wait until we have real world experience with 
pip using the sdist hook for this purpose. We don’t know what the landscape is 
going to look like once this PEP has been out in the real world for a bit, and 
we’re just trying to guess. I suspect one of three scenarios will play out:

1) We’ll decide that the build_sdist hook is good enough, and we’ll just leave 
it as is and not feel the need to do anything further.
2) We’ll decide that there are cases where the build_sdist hook doesn’t solve 
the problem adequately, and we want to add an additional hook with different 
constraints.
3) Pip will decide that the landscape in a post PEP 517 world has changed 
significantly enough to revisit our decision about how we build projects to a 
way that removes the need for a special hook in general.

Without real world experience, we don’t really know which one of (1, 2, 3) will 
be the optimal solution to this problem, and removing the hook *does* 
implicitly choose (1) for now, but (1) can be easily changed to (2) or (3) 
later on, but (2) and (3) cannot be changed to (1). So doing the simpler thing 
first gives us flexibility to adjust our solution once we have a chance to see 
how the ecosystem adjusts to this brave new world.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 5 July 2017 at 16:19, Donald Stufft  wrote:
> Quite literally, the only case I can think of that fits into this is flit’s
> “I will use git to figure out additional files, but you will have to
> configure in a static file the name of the Python package (as in import
> package) that you’re distributing and I’ll just glom down that entire
> package directory”. I know setuptools/distutils doesn’t work this way nor do
> any of the plugins that I am aware of. Best I can tell numpy.distutils nor
> enscons.

I have to say I still have deep reservations about flit's approach of
assuming/requiring that you're using VCS (git) to maintain your
project. I know that in practical terms most people will be, but it
still seems like a strong assumption to make. One of the consequences
is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
downloaded the project archive from github and unpacked that" well.
And the result of *that* is that we're putting in mechanisms in the
PEP to manage that approach.

Having said that, I think flit is a great project, and I don't think
that my personal dislike of one specific design choice is particularly
relevant here. Also, I expect flit to be an important backend, simply
because it makes it dirt-simple to package up a pure python project.
So I do think that the flit use case is important for PEP 517.

One thought - at the moment, all of the debate seems to be over the
PEP side of things. That's not surprising, as distutils-sig is for
debating standards, not tool design. But are there any changes that
might make sense for flit that could improve things? For example, add
some fallback mechanisms to flit that mean that it *can* always build
a sdist, even if it has to make guesses in the absence of a VCS (if
there's no VCS, include everything, or if there's no VCS, only include
the minimum needed to build the wheel - both seem reasonable choices,
and either seems better than "refuse to build a sdist").

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Thomas Kluyver]

On Wed, Jul 5, 2017, at 05:08 PM, Paul Moore wrote:
> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
> downloaded the project archive from github and unpacked that" well.

Flit handles these fine for everything *apart* from making an sdist. It
can make a wheel, install the package, or symlink it as a development
install. Hence why I'm so frustrated by the insistence that we must make
an sdist when we have no need for an sdist.

It's not a compromise I'm entirely happy with, but all the other options
that we came up with had bigger problems, IMO.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Donald Stufft]

> On Jul 5, 2017, at 12:08 PM, Paul Moore  wrote:
> 
> I have to say I still have deep reservations about flit's approach of
> assuming/requiring that you're using VCS (git) to maintain your
> project. I know that in practical terms most people will be, but it
> still seems like a strong assumption to make. One of the consequences
> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
> downloaded the project archive from github and unpacked that" well.
> And the result of *that* is that we're putting in mechanisms in the
> PEP to manage that approach.


I think it’s possible for us to generically handle the “I unpacked a sdist” 
case in pip by just using the current shutil.copytree approach. The bad case 
for that (large .git repositories, .tox, etc) generally only happen in actual 
development environments and not inside of a sdist itself. The only thing we 
would need is some mechanism for a front end to determine if something is an 
unpacked sdist or not.

Of course the backend could also implement it a similar way, by just tarring up 
the entire directory if it detects it’s in an unpacked sdist (which means it 
would need to add a mechanism to detect if it’s in an unpacked sdist, rather 
than us needing to add one to the PEP itself).

I don’t have a strong preference for how we do that, and we could even do both 
things TBH, and reuse Nathaniel’s idea of using the NotImplemented singleton 
and make the logic (from pip’s POV):

* From a VCS/random directory that ISNT an unpacked sdist, call build_sdist and 
if that fails or says it’s not implemented, then fail the build and surface an 
error to the user to indicate why.
* From a unpacked sdist (as determined by some hypothetical mechanism we would 
add to the PEP) call build_sdist and if that fails surface an error to the user 
to indicate why, but if it returns NotImplemented then fall back to just using 
shutil.copytree.

That would allow a tool like flit to just completely ignore the unpacked sdist 
case (other than to return a NotImplemented) and have things still work fine 
BUT it also allows another tool to do something extra in that case (such as 
verifying all of the file hashes against a record of known hashes and munging 
the version if any differ to indicate the version has changed).

That doesn’t solve the “I downloaded a archive from GitHub” or “I mounted my 
VCS checkout into a docker container without my VCS installed”, but those can 
only be solved by the backend tool itself and IMO it’s perfectly acceptable for 
the backend to fail with an appropriate error message if it needs something 
that the current directory or environment doesn’t provide.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Jeremy Stanley]

On 2017-07-05 12:40:08 -0400 (-0400), Donald Stufft wrote:
[...]
> That doesn’t solve the “I downloaded a archive from GitHub” or “I
> mounted my VCS checkout into a docker container without my VCS
> installed”, but those can only be solved by the backend tool
> itself and IMO it’s perfectly acceptable for the backend to fail
> with an appropriate error message if it needs something that the
> current directory or environment doesn’t provide.

That's exactly the approach PBR has taken since the beginning, and
it's worked just fine. Users do still wrongly assume from time to
time that they should be able to treat a "GitHub tarball" or similar
contextless pile of files like an sdist even if it hasn't been
correctly passed through the sdist build process first, but a
combination of sane fallback behaviors, environment variable
overrides and clear error messages keeps the invalid bug reports to
a manageable minimum.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Paul Moore]

On 5 July 2017 at 17:14, Thomas Kluyver  wrote:
> On Wed, Jul 5, 2017, at 05:08 PM, Paul Moore wrote:
>> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
>> downloaded the project archive from github and unpacked that" well.
>
> Flit handles these fine for everything *apart* from making an sdist. It
> can make a wheel, install the package, or symlink it as a development
> install. Hence why I'm so frustrated by the insistence that we must make
> an sdist when we have no need for an sdist.
>
> It's not a compromise I'm entirely happy with, but all the other options
> that we came up with had bigger problems, IMO.

Apologies, I should have been clearer - I did indeed mean "in order to
produce a sdist". I personally consider producing sdists and producing
wheels to be the two fundamental responsibilities of a build system,
so I dispute your statement that "we have no need for a sdist" (it's
not just pip that needs to do it, tox for example also relies on
building sdists). But this is off topic, and you've made your choice
for flit, so I'll say no more.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nathaniel Smith]

On Wed, Jul 5, 2017 at 9:14 AM, Thomas Kluyver  wrote:
> On Wed, Jul 5, 2017, at 05:08 PM, Paul Moore wrote:
>> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
>> downloaded the project archive from github and unpacked that" well.
>
> Flit handles these fine for everything *apart* from making an sdist. It
> can make a wheel, install the package, or symlink it as a development
> install. Hence why I'm so frustrated by the insistence that we must make
> an sdist when we have no need for an sdist.
>
> It's not a compromise I'm entirely happy with, but all the other options
> that we came up with had bigger problems, IMO.

What do you think of the compromise in the draft that I posted at the
beginning of this thread? The idea there is that flit would be
responsible for providing the operations "build an sdist (or say that
it can't)" and "build a wheel", and then if pip tries to build an
sdist and flit tells it that it can't, it's pip's problem to figure
out how it wants to handle that. Of course you'll still probably want
to argue with the pip devs about how they handle this, and whether
they can support in-place builds, etc., but at least those arguments
stop being blockers for PEP 517.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Nick Coghlan]

On 6 July 2017 at 07:45, Nathaniel Smith  wrote:
> On Wed, Jul 5, 2017 at 9:14 AM, Thomas Kluyver  wrote:
>> On Wed, Jul 5, 2017, at 05:08 PM, Paul Moore wrote:
>>> is that flit doesn't handle scenarios like "I unpacked a sdist" or "I
>>> downloaded the project archive from github and unpacked that" well.
>>
>> Flit handles these fine for everything *apart* from making an sdist. It
>> can make a wheel, install the package, or symlink it as a development
>> install. Hence why I'm so frustrated by the insistence that we must make
>> an sdist when we have no need for an sdist.
>>
>> It's not a compromise I'm entirely happy with, but all the other options
>> that we came up with had bigger problems, IMO.
>
> What do you think of the compromise in the draft that I posted at the
> beginning of this thread? The idea there is that flit would be
> responsible for providing the operations "build an sdist (or say that
> it can't)" and "build a wheel", and then if pip tries to build an
> sdist and flit tells it that it can't, it's pip's problem to figure
> out how it wants to handle that. Of course you'll still probably want
> to argue with the pip devs about how they handle this, and whether
> they can support in-place builds, etc., but at least those arguments
> stop being blockers for PEP 517.

Along those lines, I realised there's a variant of your "return
NotImplemented" proposal that I actually like: we can actively
encourage backends like flit that have additional requirements for
building sdists to check for those external dependencies in
`get_requires_for_build_sdist` and raise an exception if they're
missing.

That is, the "get_requires_*" hooks would have a dual responsibility:

- fail outright if external dependencies are missing
- otherwise report any Python level dependencies that the backend
wants the frontend to install

That would give us the following situation:

- as long as the build environment has the relevant VCS tools
available sdist-based builds for flit will "just work"
- plenty of build environments are already going to have VCS tools
routinely available anyway, and if they don't, adding them often won't
be that big a deal
- for the "unpacked sdist that the frontend doesn't know is an
unpacked sdist" case, a backend like flit can use either PKG-INFO or
else its own custom sdist marker file to detect unpacked sdist
directories and just tar them back up to create a fresh sdist for the
frontend to unpack

>From a frontend evolution perspective, we'd then be anticipating one
or the other of the following outcomes:

- pip gains a --build-strategy option to choose between 1) sdist based
out-of-tree builds (the default); 2) copytree based out-of-tree
builds; 3) in-place incremental builds
- we eventually decide to revise the backend interface to add back a
non-sdist based build tree preparation hook

That frontend-centric `--build-strategy sdist|copytree|in-place` idea
is starting to sound to me like it may be a better way to address both
Thomas's concern about still being able to build a wheel when the
requirements for creating an sdist aren't met ("--build-strategy
copytree" or "--build-strategy in-place") and Nathaniel's concern
about making it easy to share object files between successive builds
("--build-strategy in-place").

Recommending a frontend option like that also has a significant added
benefit over the optional backend hook: it better abides by the
principle of "In the face of ambiguity, refuse the temptation to
guess" (with the ambiguity in this case being "What behaviour will
people want if a frontend uses sdist-based out-of-tree wheel builds by
default, but building the sdist fails in a situation where building
the wheel directly would succeed?")

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] how sdist generation works in enscons

[Daniel Holth]

Here's how sdist generation works in enscons.

enscons, a build tool that exists to prototype new Python packaging
features, is just a set of tools for SCons that makes it easier to generate
wheels and sdists. If targets with certain names exist (sdist, bdist_wheel)
then the provide setup.py shim and new PEP 517 wrapper will interoperate
with pip.

For enscons itself, the sdist build rule looks like

sdist = env.SDist(source=FindSourceFiles() + ['PKG-INFO', 'setup.py',
'README.rst', 'CHANGES'])env.Alias('sdist', sdist)


Enscons would be able to build its own sdist outside a repository, and it
would include the listed files plus everything that is used to build the
other targets (the wheel).

A different package runs 'hg manifest' in a subprocess to feed to
env.SDist(), a natural thing to do in this kind of build system. That one
would not be able to build another sdist from an unpacked sdist.

Enscons itself doesn't have a way to know whether sdist generation will
succeed other than trying to run 'SCons sdist' against the user provided
build script.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] how sdist generation works in enscons

[Nick Coghlan]

On 6 July 2017 at 12:19, Daniel Holth  wrote:
> Enscons itself doesn't have a way to know whether sdist generation will
> succeed other than trying to run 'SCons sdist' against the user provided
> build script.

Right, so the most `enscons` would be able to do to help prepare the
build environment is to depend on `import-scons` (as it already does)

If the `sdist` target is then missing, or doesn't work, that's a
genuine bug in the way the project using `enscons` has been set up,
rather than something an end user attempting to build that project
would be expected to deal with themselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig