Re: [Distutils] Best way to handle packaging a project that only applies to certain versions of Python

[Daniel Holth]

Let's say you have different code for python 3.3 and python 3.4+. Tag one
wheel py33-none-any and the second py34-none-any. The second wheel is
preferred on python 3.4 and above, but ignored by 3.3. The py3 tag wouldn't
work well here.

Order of preference on 3.5 is: ('py35', 'none', 'any'), ('py3', 'none',
'any'), ('py34', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none',
'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')

Enscons would let you control the wheel tag directly but doesn't yet let
you build multiple wheels with a single invocation.

On Mon, Aug 29, 2016, 19:46 Donald Stufft  wrote:

>
> On Aug 29, 2016, at 7:34 PM, Brett Cannon  wrote:
>
> Someone has asked that I do a new release of importlib that includes a
> LICENSE file on PyPI: https://pypi.org/project/importlib/. Historically I
> have had the setup.py simply not include any Python code when built on
> versions of Python that include importlib in the stdlib itself:
> https://github.com/brettcannon/importlib/blob/master/setup.py .
>
> But now I would like to do a wheel. Is there some way I'm not thinking of
> to have a wheel that will leave out code or not install itself if a certain
> version of Python is used? Or will the user have to specify a proper Python
> requirement in their requirements.txt to get that kind of experience?
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> If your setup.py produces different output on different versions of
> Python, you’ll need multiple wheels.
>
> —
>
> Donald Stufft
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Best way to handle packaging a project that only applies to certain versions of Python

[Donald Stufft]

> On Aug 29, 2016, at 7:34 PM, Brett Cannon  wrote:
> 
> Someone has asked that I do a new release of importlib that includes a 
> LICENSE file on PyPI: https://pypi.org/project/importlib/ 
> . Historically I have had the setup.py 
> simply not include any Python code when built on versions of Python that 
> include importlib in the stdlib itself: 
> https://github.com/brettcannon/importlib/blob/master/setup.py 
>  .
> 
> But now I would like to do a wheel. Is there some way I'm not thinking of to 
> have a wheel that will leave out code or not install itself if a certain 
> version of Python is used? Or will the user have to specify a proper Python 
> requirement in their requirements.txt to get that kind of experience?
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


If your setup.py produces different output on different versions of Python, 
you’ll need multiple wheels.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Best way to handle packaging a project that only applies to certain versions of Python

[Brett Cannon]

Someone has asked that I do a new release of importlib that includes a
LICENSE file on PyPI: https://pypi.org/project/importlib/. Historically I
have had the setup.py simply not include any Python code when built on
versions of Python that include importlib in the stdlib itself:
https://github.com/brettcannon/importlib/blob/master/setup.py .

But now I would like to do a wheel. Is there some way I'm not thinking of
to have a wheel that will leave out code or not install itself if a certain
version of Python is used? Or will the user have to specify a proper Python
requirement in their requirements.txt to get that kind of experience?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Brett Cannon]

On Sun, 28 Aug 2016 at 16:55 Sylvain Corlay 
wrote:

> Distutils seems to be the de-facto standard for building extension modules
> for Python and it is used for most of the third-party extensions out there.
> I don’t think that it is reasonable to declare that it is now only meant
> for Python itself.
>

Sorry, I didn't mean to suggest distutils was suddenly going to change or
disappear, but I did want a direction/focus for what we want distutils to
become long-term as we have this discussion of what to do with distutils
any time a change is proposed. At this point distutils is practically a
black box of code as no one wants to touch out of fear of breaking the
world and that's not sustainable.


> I actually see a contradiction in pointing out some lack of involvement in
> the library to oppose well needed improvements, especially for a tool that
> has so much adoption.
>

It's not a contradiction because new features != maintenance. And the usage
of distutils makes its lack of maintenance even more scary. I guess my key
point from what I said in my previous email is that we shouldn't
perpetually ignore the fact that distutils is not to a level of support or
quality that any of us are happy with and yet so many people rely on.

For me this means we should prioritize getting our build API defined and
sdists standardized so that distutils (or a replacement) can be in the
stdlib to build Python itself only, and after that you get your build tool
from PyPI. And if that means setuptools swallows/subsumes distutils then
that's fine with me, but my key point is we are not maintaining distutils
in a way I think any of us would choose to now and so I'm trying to prod
this question to start thinking about how to fix the maintenance problem
long-term.


>
> As we discussed earlier, even though it is not a concern with C, checking
> for availability of a compiler flag becomes crucial when building
> extensions with C++ since new flavors of the language emerge every couple
> of years now. It is important to be able to output meaningful error
> messages when the compiler does not support C++[11/14/17] features if they
> are needed for a given extension. This is a new aspect of the landscape in
> this area.
>
> Finally, adding this method is a very straightforward change. `has_flag`
> simply comes aside `has_function` as a method of ccompiler. I don't see a
> more natural place for it. It would be a very weird design decision in my
> opinion to not add it there, and instead to add it to distutils.ccompiler
> by monkeypatching it in setuptools.
>

Honestly I have no comment on your feature, Sylvain, and I'm sorry your
proposal happens to be the catalyst to this discussion. I'm just trying to
get a general alignment from the PyPA group as the "distutils problem"
comes up and has the same points made every time with no general decision
on how to handle it long-term.

-Brett


>
> Thanks,
>
> Sylvain
>
> On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore  wrote:
>
>> On 28 August 2016 at 18:05, Brett Cannon  wrote:
>> > The discussion of Sylvain's proposed changes to distutils suggests that
>> > there isn't a clear-cut agreement or position of this SIG -- and thus
>> Python
>> > -- on changes to distutils, its future, etc. Is there an official
>> position
>> > I'm not aware of? If not, could we get one so we know how to handle any
>> more
>> > proposed changes to distutils?
>> >
>> > For me personally (and to start this conversation if necessary), I view
>> > distutils as the package in the stdlib that handles building extension
>> > modules in the stdlib for Python. That view means that if something
>> doesn't
>> > service that explicit goal then it shouldn't go into distutils. Now
>> that we
>> > are moving down the road towards making the build step configurable I'm
>> fine
>> > with saying that distutils is not expected to work for people other than
>> > Python itself and others can use setuptools, enscons, or other projects
>> > while we continue to improve the situation to where the build system is
>> just
>> > something you specify in pyproject.toml to generate your wheel. IOW
>> > distutils is only exposed publicly because Python doesn't hide
>> anything, but
>> > making it useful for the general case of direct usage by people is a
>> > non-goal of the package.
>>
>> FWIW, my view is:
>>
>> * distutils handles building of Python packages, both in the stdlib and
>> outside
>> * in practice, setuptools is almost always used with distutils, and
>> any proposed change to distutils could be made to setuptools just as
>> easily
>> * patches for distutils typically suffer from a lack of maintainers
>> motivated to work on them
>> * there are a few developers with a strong investment in distutils,
>> who have code that is deeply entwined with the internals of distutils,
>> and changes to distutils come with a high risk of causing issues for
>> those people
>> * 

[Distutils] setup_requires: the obvious option(?)

[Antony Lee]

Hi all,

The `setup_requires` option to `setup()` is well-known to suffer from
multiple issues.  Most importantly, as it is a keyword argument to
`setup()`, it appears too late for modules that may need to be imported for
the build to occur (e.g., Cython, for which support must explicitly
provided by setuptools itself rather than by letting Cython hook into it);
additionally, there are various contorsions that people go to to avoid some
`setup_requires` when not building the package (such as checking the value
of `sys.argv`).  `setup_requires` also uses `easy_install` rather than
`pip`, but I do not see why this could not be fixed; let's focus on the
first issue instead.

If `setup_requires` appears too late to be useful, the obvious(?) option is
to move it earlier: provide a function, say, `setuptools.setup_requires()`,
that should be called *before* `setup()` itself, e.g.:

from setuptools import setup, setup_requires
setup_requires("numpy", needed_for=["build_ext"])
try:
import numpy as np
except ImportError:
np = None
setup(..., include_dirs=[np.get_include()] if np else [])

When `setup.py` is invoked, either directly or by pip, upon the call to
`setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at
least one requirement is missing, `setup_requires()` calls asks pip to
install the required packages (similarly to `
https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and
the whole Python process replaces itself (in the `os.execv()` sense) by a
new call to `python setup.py` with this temporary directory prepended to
the PYTHONPATH.  In this new process, the arguments to `setup_requires()`
are now available and we can proceed to the rest of `setup.py`.

I feel like this idea is natural enough that someone must already have come
up with it... but I may be missing something :-)

Best,
Antony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Maurits van Rees]

Op 25/08/16 om 13:57 schreef Donald Stufft:



On Aug 25, 2016, at 7:30 AM, Nick Coghlan  wrote:

On 25 August 2016 at 06:48, Donald Stufft  wrote:

Ok, I’ve updated the PEP. You can see a diff at 
https://github.com/python/peps/pull/74/files or the entire PEP at 
https://www.python.org/dev/peps/pep-0527/ once that updated. However the 
primary differences are:

* Added Nick Coghlan as the BDFL Delegate (Thanks Nick!).
* Expanded upon rationale for dropping everything but sdist/wheel/egg.
* Allow an sdist to be either .tar.gz or .zip, but still drop all other formats.

Hopefully that can adequately address Marc-Andre’s concerns for the removal of 
the other file types, and the inclusion of .zip will address other folk’s 
concerns about losing .zip.


Thanks for those updates.

My one remaining question would be whether or not we have any projects
that are in the habit of uploading both .zip and .tar.gz for the same
release - if there are, the restriction to only one sdist per release
will need to be included in the notification message to maintainers.
(The introduction of that restriction could probably be clearer in the
PEP text as well - previously it was implied by the single extension,
but it may need to be called out more explicitly now that there are
two permitted options).


My querying shows that we have a total of 359 projects that have ever
uploaded two sdists for a single release. Looking at the number of
releases that have 2+ sdists per project, it appears like there are a
handful that do it regularly, and then the vast bulk have only done it
once or twice (see: https://bpaste.net/show/fdfce63673ec).


I miss setuptools in this list.  For the latest 26.0.0 there is a wheel, 
a tar.gz and a zip, and that has been the case for a while.


https://bootstrap.pypa.io/ez_setup.py still explicitly looks for a zip 
file.  Occasionally the package maintainers forget this and an issue is 
raised, though apparently this has been automated to avoid such a error:

https://github.com/pypa/setuptools/issues/549

That said, I occasionally have erroneously uploaded a zip file when a 
tarball for that same version was already there, or the other way 
around.  They contained potentially different code: either I forgot to 
do a 'git pull' before releasing, or the previous uploader forgot to do 
a 'git push'.  Allowing only one sdist would have prevented this.  But 
really this is our own mistake.


I think I have always discovered this and removed the new distribution 
and uploaded a full new version to avoid problems.  I don't recognize 
any packages in the list.



--
Maurits van Rees: http://maurits.vanrees.org/
Zest Software: http://zestsoftware.nl

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig