[Distutils] Re: Is ensurepip still a thing?

[Chris Barker via Distutils-SIG]

For completeness' sake: accidentally sent offline.

-- Forwarded message --
From: Chung Tzu-ping 
Date: Tue, Aug 7, 2018 at 1:54 PM
Subject: Re: [Distutils] Re: Is ensurepip still a thing?
To: Chris Barker - NOAA Federal 


Replies inline

On 7 Aug 2018, at 23:43, Chris Barker - NOAA Federal 
wrote:

>> IIRC, ensurepip by design doesn't go to the internet , so it will only
>> ever upgrade to the version bundled with Python
>
> Now I’m really confused — if pip is already bundled with Python, then
> what is ensurepip for ?!?!

Yes and no. Python does not have pip pre-installed, but distributions from
python.org comes with a wheel of pip (and one for setuptools) that allows
ensurepip to install pip withou teaching the internet. The files to make
pip work is there, but not available as a module before ensurepip is run.

The purpose of ensurepip is to make the distributed wheels available as the
importable and invokable pip module and command. This is not the only way
to make pip available for a newly-installed Python, but is the way you are
mostly encouraged to do it.

>
> Or really, the question at hand: should a user starting from scratch
> with a python.org install of 3.7 run ensurepip?

Yes. It gives you the pip command and module.

>
> Or can they just go straight to:
>
> Python3 -m pip install —upgrade pip

They can’t. pip cannot be used before you install them, either via
ensurepip or other methods (not encouraged and not discussed here).

>
> ?!?
>
> -CHB
>
>
>> from the docs "To
>> ensure the installed version of pip is *at least as recent as the one
>> bundled with ensurepip*, pass the --upgrade option" [emphasis mine]).
>> To get the latest available version, you should do `python -m pip
>> install --upgrade pip` (better than `pip install...` as it works on
>> Windows) as you mentioned.
>>
>> Paul

--
Tzu-ping Chung (@uranusjr)
uranu...@gmail.com
Sent from my iPhone



On Tue, Aug 7, 2018 at 2:00 PM, Chris Barker  wrote:

> On Tue, Aug 7, 2018 at 9:16 AM, Donald Stufft  wrote:
>
> Ensurepip is the mechanism that Python uses to bundle pip with Python. We
>> didn’t add pip to the stdlib, we added ensurepip to the stdlib. In 3.x the
>> makefiles and macOS/Windows installers will automatically run ensurepip
>> (however in both cases there are flags to disable running ensurepip— but it
>> defaults to On in 3.x).
>
>
> OK -- that makes sense.
>
>
>> > Or really, the question at hand: should a user starting from scratch
>> > with a python.org install of 3.7 run ensurepip?
>> >
>> > Or can they just go straight to:
>> >
>> > Python3 -m pip install —upgrade pip
>> >
>>
>> The intention behind ensurepip was that they could just go straight to
>> ``python3 -m pip install -U pip`` because the ensurepip was implicit when
>> they installed Python. However some downstream distributors have mucked
>> around with ensurepip in one way or another (ranging from not running it by
>> default, to out right disabling it).
>
>
> so I *think* that means that ensurepip still has a useful function, but if
> folks install python from the python.org installers (on Windows and OS-X)
> then they can upgrade pip right out of the box -- yes?
>
> -CHB
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZPRPGLYMYEWZQ2UFFUH44NIJ5VFLO7GO/

[Distutils] Re: Is ensurepip still a thing?

[Chris Barker via Distutils-SIG]

On Tue, Aug 7, 2018 at 9:16 AM, Donald Stufft  wrote:

Ensurepip is the mechanism that Python uses to bundle pip with Python. We
> didn’t add pip to the stdlib, we added ensurepip to the stdlib. In 3.x the
> makefiles and macOS/Windows installers will automatically run ensurepip
> (however in both cases there are flags to disable running ensurepip— but it
> defaults to On in 3.x).


OK -- that makes sense.


> > Or really, the question at hand: should a user starting from scratch
> > with a python.org install of 3.7 run ensurepip?
> >
> > Or can they just go straight to:
> >
> > Python3 -m pip install —upgrade pip
> >
>
> The intention behind ensurepip was that they could just go straight to
> ``python3 -m pip install -U pip`` because the ensurepip was implicit when
> they installed Python. However some downstream distributors have mucked
> around with ensurepip in one way or another (ranging from not running it by
> default, to out right disabling it).


so I *think* that means that ensurepip still has a useful function, but if
folks install python from the python.org installers (on Windows and OS-X)
then they can upgrade pip right out of the box -- yes?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UQ2KKFGW6OEJURMZABRDXE5IJIT7S6OA/

[Distutils] Re: Is ensurepip still a thing?

[Chris Barker - NOAA Federal via Distutils-SIG]

> IIRC, ensurepip by design doesn't go to the internet , so it will only
> ever upgrade to the version bundled with Python

Now I’m really confused — if pip is already bundled with Python, then
what is ensurepip for ?!?!

Or really, the question at hand: should a user starting from scratch
with a python.org install of 3.7 run ensurepip?

Or can they just go straight to:

Python3 -m pip install —upgrade pip

?!?

-CHB


> from the docs "To
> ensure the installed version of pip is *at least as recent as the one
> bundled with ensurepip*, pass the --upgrade option" [emphasis mine]).
> To get the latest available version, you should do `python -m pip
> install --upgrade pip` (better than `pip install...` as it works on
> Windows) as you mentioned.
>
> Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XGQ7HMBNLSRG37IMIQ5WXAMEDS6UNLJW/

[Distutils] Is ensurepip still a thing?

[Chris Barker via Distutils-SIG]

I'm updating some instructions for my students, in which the first thing I
do is have them run ensurepip:

$ python3 -m ensurepip --upgrade

which resulted in:

$  python3 -m ensurepip --upgrade
Looking in links:
/var/folders/ym/tj87fc850yd6526nbrn14rxmgn/T/tmpwc8nd6oj
Requirement already up-to-date: setuptools in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(39.0.1)
Requirement already up-to-date: pip in
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
(10.0.1)

(this is after a brand-new python 3.7 install on OS-X from python.org)

All good. But then I use pip, and get (after a successful install):

You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Huh? shouldn't ensurepip have updated it for me already??

Or should I simply suggest the

pip install --upgrade pip

command and not bother with ensurepip anymore?

BTW -- shouldn't that be:

python3 -m pip install --upgrade pip

to make sure they get the "right" pip?

KInda hard to keep up with the latest ...

Thanks,

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5L63NSCWKPPGVL7HKBAZGYPV2VDHYA5P/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Chris Barker - NOAA Federal via Distutils-SIG]

I agree that you are probably best off integrating with the system
packaging system in this case.

But if you do want to deploy and app with all its dependencies in a
controlled environment, conda constructor May make that easy:

https://github.com/conda/constructor

-CHB



Sent from my iPhone

On Jul 26, 2018, at 4:20 AM, Nick Coghlan  wrote:

On 25 July 2018 at 12:39, Nathaniel Smith  wrote:

On Jul 24, 2018, at 4:36 AM, Nick Coghlan  wrote:


However, there *are* folks that have been working on allowing

applications to be defined primarily as Python projects, and then have

the creation of wrapper native installers be a pushbutton exercise,

rather than requiring careful human handholding.


But it sounds like they also want to be able to install/remove/upgrade

*parts* of the Python project, for their plugin support. And maybe

upgrade the Python interpreter as well. Do any of these tools allow

that? That's the thing that really made me think about conda.


Right, that's why my suggestion was for a two layer solution (native
packaging of a base platform integration layer via fpm, combined with
pip for plugin management within that base environment), akin to the
way Linux distro packages of Firefox and Chromium still leave the
browser to do its own plugin management.

That way the fpm-built native package can depend on any required
system packages, as well as lay out the base virtual environment in
/opt. In many ways, it's the same thing that certbot-auto is already
doing, it's just replacing the current directly downloaded shell
script with native Linux packages built with fpm.

You can certainly do the same thing with conda instead (as per [1]),
but given that the target audience for certbot includes professional
Linux sysadmins, being able to integrate with the native system
package manager seems to be an actively desired feature rather than an
unwanted hassle. So while I'd agree conda is well worth a look as a
potential helper for environment management within the /opt directory,
in this particular case I don't think it's going to be a suitable
replacement for offering native packages as the core update mechanism
for the base platform.

Cheers,
Nick.

[1]
http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html#platform-management-or-plugin-management

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KJQNVRZJ4VTIO2IPAUVO4MSWDEK6WULI/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VJBMVFKAH6FYYBMHXEJSBMH5G3UQHJJJ/

Re: [Distutils] How to eliminate on part of a package?

[Chris Barker]

frankly, I'd give up n find_packages -- it's not that magic, it's just a
convenience function so you don't need to hand-specify them.

But in this case, you're doing something weird, so I"d just be explicit.

Though what I'd probably really do is make the client and server completely
separate packages. After all, you say your users only want the client side
anyway.

and if the server depends on the client (which I"d hope it doesn't!) then
you can simply make it a dependency.

-CHB




On Wed, Apr 25, 2018 at 1:28 PM, Skip Montanaro 
wrote:

> >
> > If by "top/server tree" you mean that there are more subpackages under
> top.server (not just a server.py file as your diagram shows), then you need
> to filter out all of those subpackages as well, e.g.:
> >
> > packages = setuptools.find_packages()
> > if sys.version_info.major < 3:
> > packages = [
> > pkg for pkg in packages
> > if pkg != "top.server" and not
> pkg.startswith("top.server.")
> > ]
>
> Thanks, yes, there is another subpackage within top/server, but I
> eliminated it as well. I was simplifying for the email. The raw
> find_packages() output looks like this:
>
> ['tests', 'top', 'tests.python', 'top.client', 'top.server',
> 'top.server.db']
>
> I was excising the last two elements from the returned list, so the
> argument of the packages keyword looked like this:
>
> ['tests', 'top', 'tests.python', 'top.client']
>
> Does the presence of 'top' in the list imply everything under it will
> be copied (I do want 'top', as that's the top level package, not just
> a directory in my repo.)
>
> I'll keep messing with it.
>
> Skip
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Or in version spec...

[Chris Barker - NOAA Federal]

Sorry to lose the thread — lousy iPhone mail app...

Conda supports or in its meta.yaml format:

https://conda.io/docs/user-guide/tasks/build-packages/package-spec.html#build-version-spec

Maybe look to that for prior art?

And it would be mildly less confusing to have consistency between the
systems.

-Chris


Sent from my iPhone
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outstanding questions for PEP 541: Package Index Name Retention

[Chris Barker - NOAA Federal]

> I don’t know how much we can or should define “being used”. For instance,
every package in existence is going to be downloaded via mirroring, so we
can’t go by pure download counts (and some mirrors just use pip to do their
downloading) and we’re going to need to interpret the download counts to
some degree to try and gauge “usage”.

One idea— create a couple of “dummy” packages that do absolutely nothing.
Then the download counts on those would give you a baseline for “downloaded
but not used”

-CHB

Ultimately I think the underlying question we’re answering is whether names
on PyPI are a community good or a personal good, and this PEP more or less
states they are a community good


Which they absolutely are. PyPi is a community project — the names on pypi
are just that — names on pypi. If you don’t want to be part of that
community, you can use any name you want.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [proposal] shared distribution installations

[Chris Barker - NOAA Federal]

, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.


For what it’s worth, conda environments use hard links where possible, so
limiting duplication...

Maybe conda would solve your problem...

-CHB







I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:

Hi Ronny,


What you describe here is, as you know, basically what the Nix

package manager does. You could create something similar specifically

for Python, like e.g. `ied` is for Node [2], or Spack, which is

written in Python. But then how are you going to deal with other

system libraries, and impurities? And you will have to deal with

them, because depending on how you configure Python packages that

depend on them (say a `numpy`), their output will be different. Or

would you choose to ignore this?


Freddy


[1] https://nixos.org/nix/

[2] https://github.com/alexanderGugel/ied

[3] https://spack.io/


On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt  wrote:

Hi everyone,


since a while now various details of installing python packages in

virtualenvs caused me grief


a) typically each tox folder in a project is massive, and has a lot

of

duplicate files, recreating them, managing and iterating them takes

quite a while

b) for nicely separated deployments, each virtualenv for an

application

takes a few hundred megabytes - that quickly can saturate disk

space

even if a reasonable amount was reserved

c) installation and recreation of virtualenvs with the same set of

packages takes quite a while (even with pip caches this is slow,

and

there is no good reason to avoid making it completely

instantaneous)


in order to elevate those issues i would like to propose a new

installation layout,

where instead of storing each distribution in every python all

distributions would share a storage, and each individual

environment

would only have references to the packages that where

"installed/activated" for them


this would massively reduce time required to create the contents of

the

environments and also the space required


since blindly expanding sys.path would lead to similar performance

issues as where seen with setuptools/buildout multi-version

installs,

this mechanism would also need a element on sys.meta_path that

handles

inexpensive dispatch to the toplevels and metadata files of each

packages (off hand i would assume linear walking of hundreds of

entries

simply isn't that effective)


however there would be need for some experimentation to see what

tradeoff is sensible there


I hope this mail will spark enough discussion to enable the

creation of

a PEP and a prototype.



Best, Ronny




___

Distutils-SIG maillist  -  Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[Chris Barker - NOAA Federal]

 distutils works fine for its original purpose (building components for
 the system Python in Linux distros),

What does Linux have to with it? In the eagle days, I found it most
helpful for Windows, actually. And it's very helpful for OS-X as well.

It was also great for pure python bundling and installing.

But regardless of original intent, it's clear that distutils has not
turned out to be as cleanly extendible as is needed.

Setuptools has its issues as well -- personally it has always driven
me crazy with its mingling of run-time vs build-time vs install time
functionality.

I think it would be helpful to be explicit about the final goals
before talking about whether and how to fork distutils.

What I'd love to see is a build tool that is compatible with the
"modern" packaging paradigm-- pip, pypi, wheels. ( and other systems
like rpm, conda, )

Essentially setuptools without the cruft. ( no easy install, pkg_resources, ...)

This could be maintained as a third party package, like the current
setuptools. If/when it becomes ubiquitous, an "ensure" style module
could be added to the stdlib.

So how do we get there? I would think a fork and merge of both
setuptools and distutils would make sense. Preserve the building part
of the API, so folks could simply replace the import line an their
setup.py files to use the new tool.

Distutils and setuptools remain in maintainable only mode for probably
a long time :-(

-CHB



 so we still need to avoid
 breaking that. setuptools is only essential if you want full support
 for modern *Python* level packaging features (PEP 376 install
 metadata, venv compatibility, wheel files, etc), and a lot of Linux
 system components simply don't worry about those things, and rely on
 their system level equivalents instead (e.g. the RPM/deb databases,
 chroots and containers, RPM/deb files)

 However, what *could* be interesting is a proposal to move distutils
 to the "ensurepip" model, where rather than maintaining distutils
 directly as part of CPython, the CPython build process instead runs
 setuptools/distutils from a bundled wheel file. Doing that would
 entail having setuptools actually start installing a copy of distutils
 into site-packages: older CPython releases would ignore it by default
 (since the stdlib version would shadow it), while 3.7+ would offer
 either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools"
 (bikeshed to be painted via the PEP process) to install the
 setuptools-provided version.

 I'm not claiming actually doing that would be particularly easy - I
 just think it's the most viable path to get us away from the current
 version coupling between the build infrastructure in distutils and the
 runtime support infrastructure in the rest of the standard library,
 and to avoid maintaining two distinct copies of distutils indefinitely
 (one in the stdlib, one in setuptools).

 That approach wouldn't even entail any *new* bundling at the CPython
 level, as while it's currently formally an implementation detail
 (pending potential removal in a post-PEP-517 world), setuptools is
 already bundled as part of the support infrastructure for ensurepip:
 https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accepting PEP 541?

[Chris Barker - NOAA Federal]

This looks great: thanks for moving it along!


Minor notes (all copy editing):

It could use some editing to bring it into the present:

"""
Existing package repositories such as CPAN [3]
, NPM [4]
, and GitHub [5]
 will be investigated as
prior art in this field.
"""

Probably need a time scale on:


   - no activity from the owner on the project's home page.

"""

Minor edit:
"the candidate is able to demonstrate own failed attempts to contact the
existing owner"

To

"the candidate is able to demonstrate their own failed attempts to contact
the existing owner"

(Adding "their" -- or his/her)

Same under "abandoned".


-CHB

Sent from my iPhone

On Sep 8, 2017, at 12:14 PM, Lukasz Langa  wrote:

I gathered and responded to feedback on this early this year. I think the
PEP is ready to be accepted.

https://www.python.org/dev/peps/pep-0541/

- Ł

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Sep 4, 2017 at 6:00 AM, Nick Coghlan  wrote:

> >> Do the Linux distros use pip to build their packages?
> >
> > Not that I am aware of.
>
> Fedora's build macros for Python projects currently rely on running
> setup.py directly, but we've been considering switching to pip instead
> since 2013 or so. PEP 517 is likely to provide the impetus to switch
> from "maybe we should do that" to "we need to do that, at least if
> setup.py is missing, and potentially always, so we get more consistent
> installation metadata"
>

which is exactly why I tried to do it in conda. In a post PEP 517 world,
that may be a good way to go. I'm looking forward to it.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:

since it
> doesn't reliably distinguish between "this cached wheel was downloaded
> from a repository" and "this wheel was generated locally with a
> particular version of Python".


It shouldn't have to. sigh.

>
 PEP 517 deliberately doesn't let
> frontends do that as part of the initial build process (instead, if
> they want to adjust the tags, they need to do it as a post-processing
> step).
>
> Since PEP 517 breaks the current workaround for the caching scheme
> being inaccurate, the most suitable response is to instead fix pip's
> caching scheme to use a two tier local cache:


I'm still confused -- if setuptools ( invoked  by pip) is producing
incorrectly named wheels -- surely that's a bug-fix/workaround that should
go into setuptools?

If the build is being run by pip, then doesn't setuptools have all the info
about the system that pip has?

I'm trying to think through the use case presented the other day:

A package can be build either for all python's or for cpython in
particular. That selection is (currently) done at build time, by custom
code in setup.py.

In this case, the build system SHOULD be able to build the proper wheel,
and label it correctly.  But there might need to be a way to tell the build
system which you want. That logic isn't in existing setup.py files, but
there is only so much you can do!

Then there is the caching problem. If pip finds a pure-python wheel that
matches the spec in the cache, it won't know to go looking for a cpython
specific one. I can't see how separating the downloaded  cache from the
locally built cache will solve that problem:

If a pure-python wheel was already downloaded, then that'll be in the
downloaded cache anyway.

I think the core problem here is having, for instance, pypy and cpython
sharing a wheel cache. Is it really that important to support that?

If so, this is a total kludge, but maybe the pip cache could keep a
record-- I looked for a cpython specific wheel, and it doesn't exist, so I
never need to look again.

Or -- maybe more kludgy -- every time pip puts a  wheel in the cache via
cpython, it makes a link with the cpython name.

Independently of that though, it's also important to note that pip
> probably *won't* switch to invoking legacy setup.py files through the
> PEP 517 backend - instead, it will only do that if there's a
> pyproject.toml file that says "Use the setuptools PEP 517 backend".


That's probably a good way to keep backward compatibility while moving
forward. Which means pip can have two modes -- legacy mode and pep517 mode.

Which may encourage me to stop being such a noodge -- As long as we're
talking legacy mode, we can be as hacky as we need to be.

Autobuilding support definitely *won't* be removed from pip,


I'm not suggesting it should be (and I think I may be the only one that
might have been) However, I do think auto-building support does not have to
support all the complex use cases.

we also have plenty of PyPI users that
> explicitly *opt out* of using publisher-provided pre-built binaries.
> While Linux distributions are the most common example (see [1] for
> Fedora's policy, for example), we're not the only ones that have that
> kind of rule in place.


But this is an argument for why pypi should host sdists, and the build
tools should build sdists, but not why pip should auto-build them.

>
Condo-forge, for example, almost always builds from source -- sometimes an
sdist
from pypi, sometimes a source distribution from github or wherever the
package is hosted. And sometimes from a git tag ( last resort).

Do the Linux distros use pip to build their packages?

I tried to do that with conda-packages, and failed due to pip's caching
 behavior-- it probably would have worked fine in production, but when I
was developing the build script, I couldn't reliably get pip to ignore
cached wheels from previous experimental builds.

So I ( and the entire community, as far as I can tell) stuck with calling
setuptools directly - via setup.py, but with setuptools-specific flags)

As pip is a package management tool, it makes sense that other package
managers wouldn't need to use it.

NOTE: I do use pip to build conda packages from wheels in a couple cases --
but that's considered a work around for my pathetic build skills -- and
even then, I don't have pip download the wheel -- that's conda-build's job.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

One thing to keep in mind is that there are quite a few projects on pypa
with pure python source distributions uploaded that will not be updated and
people may still desire to use. We want pip to be able to still build and
install them.


That is the challenge!

But the motivating use case here was a package with a c extension that was
optionally built.

That's the rare case.

Regular old pure-python packages are the usual case, and a lot easier.

-CHB

2017-08-31 16:29 GMT-05:00 Chris Barker :

> On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:
>
>> > Surely the build system should know how to correctly name the wheel it
>> builds.
>>
>> It's probably worth mentioning the specific problem that motivated pip
>> to start doing this.
>>
>> It used to be standard, and is still quite common, for setup.py
>> scripts to contain stuff like:
>>
>> install_requires = [...]
>> if sys.version_info < (3, 4):
>> install_requires += [...]
>> if platform.python_implementation() == "PyPy":
>> install_requires += [...]
>>
>> setup(..., install_requires=install_requires)
>>
>> This kind of logic in setup.py worked fine in the old days when all
>> you did was 'setup.py install', but then wheels came along
>
>
> And indeed, setuptools originally used easy_install, which was part of
> setuptools...
>
> and
>> retroactively turned lots of setup.py scripts from working into
>> broken. The problem is that with this kind of setup.py,
>>
>
>
>> But it will take a while for existing setup.py files transition to
>> using those, and in the mean time pip can't assume that a random wheel
>> generated by 'setup.py bdist_wheel' has accurate Python tags.
>>
>
> This was my original point -- I understand that we want "pip install" to
> continue to work for, hopefully, everything it works for now.
>
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.
>
> Sorry to be poking at all this from the fringes (Not having been all that
> involved in a very long discussion), it's just that the whole
>
> distutils--setuptools--pip--distribute--setuptools--pip
>
> stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
> backward compatibility may end up leading us to another poorly de-coupled
> design.
>
>
>> Hopefully new legacy-free backends will get this right from the start.
>>
>
> exactly -- let's keep the "backward compatibility hack" labels clear!
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:

> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along


And indeed, setuptools originally used easy_install, which was part of
setuptools...

and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py,
>


> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>

This was my original point -- I understand that we want "pip install" to
continue to work for, hopefully, everything it works for now.

But I do think we should be clear about what is a hack for backward
compatibility, and what is part of the designed functionality.

Sorry to be poking at all this from the fringes (Not having been all that
involved in a very long discussion), it's just that the whole

distutils--setuptools--pip--distribute--setuptools--pip

stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
backward compatibility may end up leading us to another poorly de-coupled
design.


> Hopefully new legacy-free backends will get this right from the start.
>

exactly -- let's keep the "backward compatibility hack" labels clear!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft  wrote:

> I'm a bit confused -- are we talking about the backwards compatible
> path to the future -- or the end-game?
>
>
> Pip purposely overrides what setuptools tags the wheel with in order to
> make it extremely specific to the Python it is currently being run on. Now,
> a big part of why it needs to do this, is because a lot of setup.py were
> not written with wheel in mind, so they produce broken wheels by default.
>

right -- just so we're clear that this is hack to maintain backward
compatibility. Though that seems like a hack that setuptools should have,
not pip.


> One such example is a setup.py that produces a pure Python wheel for PyPy,
> and a C-extension wheel for Python. If someone installs that project on
> PyPy first and they only have a sdist available on PyPI, then pip will
> cache a pure Python wheel for that project, and by default (without this
> bit to force it) that wheel would be acceptable for CPython too and would
> have been used instead of building a wheel with a compiled extension for
> CPython.
>

Frankly, I'm not sure that pip should be sharing a cache between PyPy and
cPython anyway..

(I've had other issues with pip's fairly aggressive caching in the past...)

But if it does, then pip should always look to see if there is a more
"specific" version of a package available anyway.

But I see your point -- if there is a cached pure-python wheel, pip can go
look on PyPi for a cpython wheel, but if it isn't there, and there is only
an sdist, it would have to download and build it to find out if there is a
cPython version. Which of course, would then happen (and be wasted effort)
 essentially all the time, as most packages are pure python.

But the proper solution to this is for packages that have both a pure
python and cpython version available to put the wheels up on pypi. After
all, this is the unusual case.

In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for
distribution.

-CHB











> You could argue that the new crop of build backends should handle this
> better— and hopefully you’re correct— but I think that the two desires are
> at odd with one another. Most build backends are going to want to, by
> default, tag the wheel with as general of a tag as they think will work
> given the information available to them, whereas with pip’s internal wheel
> cache, we generally want that wheel to be as specific to the current
> version of Python as possible.
>
> Another possible solution is to stop using the wheel tagging format to
> encode this information for pip’s internal wheel cache and to encode it
> into the path, so that instead of having PyPy and CPython share a cache
> directory, they each have their own. That isn’t an unreasonable mechanism
> for doing that and which one gets used is an implementation detail of pip.
>
> —
> Donald Stufft
>
>
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore  wrote:

> > As to using pip to build wheels -- there is good reason to do that
> > now, but in s post PEP 517 world, one would call the build system
> > directly to build a wheel-- after all, all pip should be  doing is
> > calling the build system anyway.
>
> I disagree - "pip wheel" will still be useful, for example to obtain a
> wheel from PyPI either by downloading or download & build. Also just
> to have a unified interface that works regardless of the project
> backend - if a project switches from setuptools to flit, for example,
> it would be good if deployment and test scripts didn't have to change.
>

Isn't that why we have PEP 517? I unified interface to build tools?

So I still expect pip wheel to be useful in a post-PEP 517 world.


Maybe so -- but all pip should be doing is passing off the job to the
back-end.

Again, the package manager, well, manages the packages. It shouldn't be
concerned with how to build them.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

> that neither pip nor the setuptools backend should not change the tags
> it applies to wheels by default).

I'm a bit confused -- are we talking about the backwards compatible
path to the future -- or the end-game?

In short -- I'm sure we'll have to do some hacky stuff to keep
backwards compatibility, but the end game should be a clean separation
of concerns :

Pip (or any front end) should never "build a wheel", and it certainly
shouldn't have to know enough about what's in a wheel to be re-naming
it for generic python vs cpython.

The package manager should manage the package, not built it, or change it.

Surely the build system should know how to correctly name the wheel it builds.

As to using pip to build wheels -- there is good reason to do that
now, but in s post PEP 517 world, one would call the build system
directly to build a wheel-- after all, all pip should be  doing is
calling the build system anyway.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

> conda-build doesn't produce or consume wheels.  It works by creating a
> clean
> > conda environment and running a shell script to install the python
> package
> > into that environment


To be really clear -- conda build doesn't directly use ANY build/install
system.

All files produced by the install
> > script are then put into a tar file, which is conda's version of a wheel.
>

right -- conda-build simply runs the script it is told to run (by the
recipe), and then looks and sees what files were installed.

In theory, one could build a conda package directly out of a binary wheel
-- after all, as mentioned, they are actually pretty similar, but most
folks would probably just use pip inside their conda-build recipe.

In fact, that is already being done -- most python packages are built with
"setup.py install", but some are built by using pip to install a wheel.

But I didn't think that the goal of PEP 517 was to ensconce pip as the only
way to install a package in Python.

So if a build system can install (like setuptools currently does), then
conda recipes may well call it directly.

I don't think this really has any bearing on the the PEP, though -- As I
understand it, the goal of the PEP is to formalize the interface between
build systems and package managers -- and that can only be a good thing for
other package managers.


NOTE: For the most part, folks want to use a pip-compatible install in
conda-build, because users tend to also directly use pip to install stuff
in conda environments e.g. if there is not a conda package available. If a
conda package were built with, say, plain distutils, then pip won't find
the meta data it's looking for, and might falsely see an unresolved
dependency.

Personally, I've always avoided pip installing packages in conda --
preferring to make conda packages for all the deps I need, but conda-pip
compatibility has come a long way, so it does work fine for many users.

But it would be nice if packages didn't NEED to be pip installed -- but I
think that's an issue with the wheel spec (that may well be solved).

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore  wrote:

> Me too. At the moment, I only expect two backends of any substance -
> your flit backend and xoviat's setuptools one. But I only know of one
> frontend, namely pip - and talk of projects like tox or twine acting
> as frontends never seems to get any traction.
>

If the build API is clean and supported enough, I can see conda using it.

just sayin'

NOTE: it wouldn't be conda per-se using it -- but individual conda recipes,
which generally call setuptools directly now.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 12:20 PM, Paul Moore  wrote:

> > interface). So if pip calls build_sdist and then build_wheel, there will
> be
> > two source distributions built (one by pip and one by setuptools) before
> > building a wheel.


exactly why pip should NOT concern itself with the sdist -- either the back
end needs to do it anyway, or it doesn't need to do it at all, and would
only be doing so to satisfy pip.


> This is precisely the question - whose responsibility is it to ensure
> that sdist->wheel vs build_wheel equivalence is maintained? The PEP
> says it's the backend's responsibility, so pip shouldn't need to do
> anything.
>

makes sense to me.

But, we're currently having debates about "return None" which hinge on
> "if backends have bugs, return None might not work well". Sure - but
> if backends have bugs, they may not maintain wheel equivalence.


If backends have bugs (and they will), any number of things can go wrong.
Making the spec so that front ends can have a better idea what went wrong
is a fine idea, but it can only go so far.

The only thing we really need to distinguish is " that functionality is not
implemented" from "something went wrong"

And not even that if we don't have pip trying to go through some
machinations about what do if the back-end does not produce a sdist.


>  To that extent, I'm with Donald - pip going
> sdist->wheel protects the user against the known-to-be-an-issue bug
> that the backend doesn't ensure wheel equivalence.


pip can not protect the user from a poorly written back-end. And it
shouldn't try.


* Chris Barker has pointed out that backends have no reason to support
> sdists now.
>

not quite -- the reason to support sdist is because you want an sdist, not
because it's a necessary part of the path to a wheel. I don't quite
understand your (Paul's) point about the importance of sdists to open
source (as opposed to a plain old tarball of the source, like what gitHub
produces when you do a release.

* Nathaniel is pushing a means of notifying "I can't build a sdist"
> that protects against backends accidentally not following the spec.
>

"I can't do that" respond to any possible request makes sense. Unless "I
can't build a sdist" violates the protocol -- which I don't think it should.

> Should we trust the backend or not? Backends *will* have bugs - part

> of "trust the backend" is simply telling the user that the problem
> behaviour they found is not pip's issue and should be reported to the
> backend. Is that a sufficiently bad user experience that we should try
> to improve it (experience with setuptools says that it is - why are we
> assuming that developers of new backends will be so much more
> conscientious and careful than the setuptools developers? *I*
> certainly don't feel that the setuptools developers are unusually bad
> - quite the opposite!)
>

If that is a sufficiently bad user experience then the only other option is
for "us" by some definition of "us" to build the back-end, too -- which is
where this all started with setuptool. But my understanding is that goal is
was not to force the whole stack to maintained together.

And really, the end-user will need to report the problem to the package
maintainer, not the build tool maintainer jsu tlike they should no if
something doesn't pip install.

Having a more robust means of saying "I can't build a sdist" than
> "return None" is protecting the user from issues with the backend. So
> is building via sdist.
>

but only one particular set of issues -- when there are an infinite number
of possible issues...

One other thought --

the easiest way to make an sdist it to simply copy the source tree.

So:

1) not a big deal to require back-ends to do it -- it's easy to do
but
2) back ends may do it the easy and sloppy way, and get build artifacts in
the sdist, and then you are back where you started.

so requiring the sdist step in no way saves you from poorly written
back-ends. It may make it worse, as you _think_ you are getting something
clean.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py

[Chris Barker]

I've thought for ages that we could  transition to a more sane system by
convention:

"your setup.py, after being imported, will have a "setup_params" attribute
that is a dict that can be passed to setup()."

So tools that want metadata, etc. without actually running an install could
do;

import setup

name = setup.setup_params['name']
requirements = setup.setup_params['install_requires']

etc...

i.e -- it does need to run the code in setup.py, but it does NOT need to
run setup install, or build, or whatever else.


A typical simple setup.py with static meta-data would be:

from setuptools import setup

setup_params = {name:'sample',
version:'1.2.0',
description:'A sample Python project',
url:'https://github.com/pypa/sampleproject',
author:'The Python Packaging Authority',
license:'MIT',
packages:['the_package'],
install_requires=['peppercorn'],
...
}

setup(**setup_params)

But a setup.py could still have arbitrary code to build up the dict of
parameters if need be.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft  wrote:

> Donald, what do you think? IIRC, you were most keen on going
> sdist->wheel where possible, and I don't think you've commented on
> Paul's suggestion yet (apologies if I've overlooked a response).
>
>
> I still think it should, and prefer pip to attempt to build a sdist prior
> to building a wheel when we’re coming from a VCS directory. I think that is
> going to be the most robust mechanism with the least amount of surprising
> behavior for end users.
>

why? even as it stands, doesn't setuptools dump everything into the build
dir anyway? The "Creating of an sdist" really seems like a build-system
problem, not a package manager problem to me.

> "when we’re coming from a VCS directory."

whether this is a VCS directory or some other source seems to me like
something pip should not need to know...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Sun, Aug 27, 2017 at 2:59 AM, Paul Moore  wrote:

> > Suppose you
> > have a frontend like pip that when given a source directory and asked
> > to build a wheel, wants to implement that as:
> >   - build sdist
> >   - unpack sdist
> >   - build wheel from unpacked sdist
>
> ... I'd like to
> point out that in
> https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html
> I confirmed that I'm basically persuaded now that pip should just
> "trust the backend" and use build_wheel directly. So this whole debate
> isn't relevant for pip.


Nor should it be for any other system. Before I saw this post I was about
to make a point about not contorting ourselves to make things works the
same way as pip and setuptools currently work -- pip was kind of hacked
together to support setuptools, and setuptools grew kinda-organically from
distutils (while also shoehorning in other not-really-building
functionality).

So let's make the "new" API clean and adapt the current tools to that.

Specifically, I'm really confused about the sdist concept -- sure, it's a
nice idea to have a standard way to create a source distributions, and I
can see that tools that, for instance, create uploads for PyPi and the like
will want to do that, but I don't see it as an inherent part of building a
binary distribution (i.e. wheel), or building and installing a package
directly.

So:

If you want a wheel, the build system should be asked to build a wheel.
If you want an sdist, the build system should be asked to build an sdist --
and it can politely decline.

Whether the build system first makes an sdist, and then builds a wheel from
that it entirely up to the build system.

Is there any reason to make this more complicated?

We'll need to wait for other
> currently-hypothetical frontends like tox or twine which might use the
> build_wheel hook, before we'll have any practical experience to
> confirm if there's an issue with "return None" as the way for backends
> to signal that they weren't able to build a wheel.
>

Sure -- though I'm with Nathaniel on this one (I think -- kinda hard to
keep th threads straight!) getting a NOne can happen for alot of reasons,
better to have a more information-rich way to know that something failed.

And isn't "be able to build a wheel" a requirement for ANY supported
back-end? In which case, failure to build one is a failure that could have
been caused by any number of issues -- shouldn't it simply fail with an
Exception -- and the back end will hopefully generate nice exceptions for
the common failures?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distuils MSVC link failure between libraries

[Chris Barker - NOAA Federal]

I'm trying to create multiple C++ extensions that have dependencies between
them.

This works fine on Linux with gcc, but I get link failures on Windows with
MSVC due to this



Is that the only issue? MSVC and FCC linking is pretty different-- I know I
tried to do something like this (a few years ago) and it didn't work -- I
think because the symbols I needed weren't exported, or something like that.

If precompiled extensions are to be published with MSVC and expected to be
linked against (maybe this is not recommended?)


Indeed, it is not recommended-- onhe passive sense -- I.e. the system isn't
designed to support that.

IIUC, while extension modules are dlls, they are specifically designed to
be called from cpython, not as general purpose libraries to be called from
other C code.

The solution is to build a dill with all the C/C++ code you need, and then
have all your modules link against that dll.

Proof of concept



If that works, and doesn't cause other problems, (I am not qualified to
assess that), it could be handy. There are times I would have liked
distutils to be able to build a dll for me that could be used by other
modules.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] new PyPI: a rant from a package maintainer

[Chris Barker]

On Fri, Aug 4, 2017 at 5:42 PM, Lucas Boppre Niehues 
wrote:

>
> The long description was originally Markdown, and converted to RST by
> pandoc. I would 100% understand if this conversion triggered some bug.
>

It's a good idea to run docutils on it yourself -- but in any case, broken
RST has always resulted in plain text rendering on PyPi in the past -- as
it should.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] new PyPI: a rant from a package maintainer

[Chris Barker]

one point:


I probably should have named it something other than /legacy/,
>

yes, it should have -- names matter! and having a "legacy" in teh name when
there is not "modern" or "current", or, indeed, anything else is very
confusing.


 pypi.org, might as well get it all done at once. It might be reasonable to
> name it something else now, and just keep the /legacy/ around as an alias.
> I’m not sure if that adds or subtracts from the confusion, but if you think
> that would have helped you, please open a new issue on Warehouse.
>

I think that would be better -- something like "version1" something would
be more clear.

However, we have a major jumble of semi-out of date documentation scattered
all over teh internet -- so ANOTHER name change, might just make things
worse -- it's also confusing to have two names for the exact same thing!

tough call.


> HTTPError: 400 Client Error: Invalid file extension. for url:
https://upload.pypi.org/legacy/

The ability to upload anything besides sdists, wheels, and eggs was
deprecated and removed.

A nice message to that effect would be great.

Also -- I know that the various tools involved are maintained by different
people, but it sure would be nice if, for instance, the latest versions of
setuptools and twine would not try to upload non longer supported file
types!

I've thought for a couple years that we need a setuptool-lite that would
ONLY do what we think is really should do -- but no one ele seemed to think
that would be useful.

> Maybe the tutorial is outdated, and TestPyPI supports auto-registration
now?

I don’t think anyone has kept the tutorial on wiki.python.org up to date.
To be frank, I don’t even know how to update wiki.python.org.

Someone that does really needs to add a "this isn't updated, please see
packaging.python.org note on there !!

As far as keeping PyPI running as well as getting the new code base
> developed and deployed… that’s about it [2]. This is a service used by
> ~everyone in the Python community without even a single full time person on
> it.


This is a core service -- we really need to figure out a better way to keep
it maintained. I suspect the issue is project management, rather than
volunteers -- there are a heck of a lot of talented folks that depend on
these services.

Of course, I have no good ideas about how to move forward -- is there
anyone at the PSF that could take leadership on this?

-CHB
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Which commercial vendor?

[Chris Barker]

On Thu, Apr 6, 2017 at 5:34 PM, Wes Turner  wrote:


> Chances are, there will be a package or two that you rely on that is not
>> in conda defaults (maintained by Continuum) or currently in conda-forge. So
>> you can pip-install those few -- but what if they aren't on PyPi either? or
>> are hard to compile and install with ugly dependencies?  You can contribute
>> build recipes to conda-forge, and then have it for you, and all your users,
>> and the rest of the world to access. Much better than hand maintaining
>> stuff yourself.
>>
>
> Someone still needs to commit to maintaining the conda package; otherwise
> who knows whether this is the latest  stable  release?
>

Indeed. and it it's a not-that-widely-used package, then you will have to
do that yourself -- but using the conda-forge infrastructure makes that
(relatively) easy. In contrast -- who knows whether the package on PyPi is
the latest stable release? Hopefully the maintainer is keeping it up, but
if not, you're kinda dead in the water.


>> My pain point now is still full multi-platform support. conda has package
>> versions that are platform independent, but it can still be hard to get
>> everything built  in the same version on all platforms, so it does get a
>> bit ugly.
>>
>
> Docker images are reproducible and archivable:
>

In a way Docker, as I understand it, is orthogonal to this conversation.
And when I talk about "all platforms", I mean running natively on all
platforms -- I can't give my Windows users a Linux VM and expect them to
know what the heck to do with it.

Not that Docker isn't a really useful tool to htlep address some of these
problems...

-CHB



> What platforms does conda-forge auto-build for?
> - [x] x86[-64]
>

 linux-64
 win-32
 win-64
 osx-64

(all Intel)



- [ ] linux-armv7l
>   - https://github.com/conda-forge/conda-forge.github.io/issues/269
>


looks like folks are trying, but it's not really there yet -- mostly due to
the lack of easy to access CI services for armv7I

It's an open-source project -- if it's important to someone, it will get
done.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Which commercial vendor?

[Chris Barker]

On Wed, Apr 5, 2017 at 6:41 PM, Nick Coghlan  wrote:


> PayPal Engineering put together a decent write-up of their path
> towards adopting that model last year:
> https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/


Thanks for that link.

We're a much smaller shop, but have had pretty much the same experience --
also really good to see them mention miniconda at the end -- I think that's
a much better way to go for most folks than the whole Anaconda pile.

It's definitely a reasonable way to go for organisational
> infrastructure, but even conda doesn't cover all the potential use
> cases that are out there


of course not -- nothing does, but I would add to the contents of that post:

The conda-forge project is a Major boon to the conda infrastructure --
there is now a robust way for the community to expand the available number
of packages (and keep up more recent versions). And anyone can take
advantage of that infrastructure for their own (selfish?) needs:

Chances are, there will be a package or two that you rely on that is not in
conda defaults (maintained by Continuum) or currently in conda-forge. So
you can pip-install those few -- but what if they aren't on PyPi either? or
are hard to compile and install with ugly dependencies?  You can contribute
build recipes to conda-forge, and then have it for you, and all your users,
and the rest of the world to access. Much better than hand maintaining
stuff yourself.

My pain point now is still full multi-platform support. conda has package
versions that are platform independent, but it can still be hard to get
everything built  in the same version on all platforms, so it does get a
bit ugly.

But no other solution makes that better anyway.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Which commercial vendor?

[Chris Barker - NOAA Federal]

> On Mar 30, 2017, at 1:53 AM, Thomas Güttler  
> wrote:
>
>
> My frustration has reached a limit. Yes, I am willing to pay money.
>
> Which vendor do you suggest to give me a reliable package management?

You may want conda -- it's an open source project, and you can get
commercial support through Continuum Analytics.

Though conda, and Continuum was born in the data science / scientific
computing world, there is nothing about conda specific to data
science. It's just that computational computing poses extra challenges
for packaging -- but the easy stuff is still easy.

And the conda-forge project provides an open source way to support a
larger community.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] install questions and help requested. ---pyautogui

[Chris Barker - NOAA Federal]

This is really a list for discussing development of distribution
tools, rather than help on basic usage. But;

>
>  >>> pip install pyautogui
> SyntaxError: invalid syntax

This looks like you are trying to run pip at the Python prompt. Pip is
designed to be run st a system command line ("DOS prompt")

Try the same command there.

-CHB


>
> Sincerely,
> Michael G. Strain  Jr.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Chris Barker]

On Fri, Dec 16, 2016 at 5:51 AM, Daniel Holth  wrote:


> One possibility to consider is that virtualenv itself is a bad idea. Why
> should the Python interpreter executable, rather than the program being
> run, determine the set of packages that is available for import?
>

well, way back when, som eof us suggestted that pyton have pacakge version
mangement built in to import:

import this_package>=2.1

or whatever.

At that time, the pyGTK and wxPython projects had done a role-your-own
version of this. wxPython's was:

import wxversion
wxversion.select('2.3')
import wx

kind a kludgy, but it worked.

However, Guido, among others was pretty adamant that this was NOT python's
responsibility.

Then, along came setuptools that kinda-sorta provided something like that,
and then virtualenv -- and the rest is history.

I now use conda, which provides environments that manage python itself,
other C libs, etc, and it works pretty well.

And there are utilities that let you run a script in a given environment:

https://github.com/pelson/conda-execute

(and maybe others)

So that does kinda pass the responsibility to the app.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Chris Barker]

On Thu, Dec 15, 2016 at 8:29 PM, Glyph Lefkowitz 
wrote:

> At the beginning of your story you mentioned the GUI client - *that* is
> the missing piece ;).  I've been saying for years that we need a Python.app
> that lets you easily bootstrap all this stuff: walk you through installing
> C dev tools if your packages need them, present a GUI search interface to
> finding packages, present a normal "file->open" dialog for selecting a
> location for a new virtualenv, automatically pop open a terminal, launching
> a Jupyter notebook whose kernel is pointed at said environment...
>

Anaconda provides something like this -- personally, I;m a command lien
geek, so have no idea how much or if it's any good. But might be worth a
look if you're interested.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Chris Barker - NOAA Federal]

>
>
> I think it's unfair to describe these efforts as a "kludge";

I was specifically referring to using wheels to deliver C libs--
pip+wheel were not designed for that. But I don't mean to offend,
there has been a lot of great work done, and yes, the situation has
much improved as a result.

There are times when a kludge is called for!

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Chris Barker]

As pointed out by others, there are external groups doing "curating".
conda-forge is one such project, so I'll comment from that perspective:


It's difficult because the definition of compatibility is highly dependent
on
>
> the consumer's environment.  For example, C extension compatibility will
> depend on the version of libraries available on the platform versions you
> care
> about.


Indeed -- which is why Anaconda and conda-forge are built on conda rather
than pip -- it is designed to handle these issues.

However with the many linux effort, and some efforts to kludge C libs into
binary wheels, pypi may just be able to handle more of these issues -- so
curating may have it's advantages.

As it happens, I recently proposed a version system for conda-forge (much
like Anaconda) where you would select a given conda-forge version, and ALL
the packages in that version would be expected to work together. the idea
is that you don't want one package dependent on libjpeg6 while another
depends on libjpeg7, for instance.

But the community has decided that rather than try to version the whole
system, we instead rely on robust "pinning" -- i.e. version specific
dependencies -- package A depends on libjpeg7, not just "libjpeg". Then
there are tools that go through all the packages and check for incompatible
pinnings, and update an build new versions where required (or at least ping
the maintainers of a package that un update is needed)

Ubuntu has an elaborate automated system for testing some dimension of
> compatibility issues between packages, not just Python packages.  Debian
> has
> the same system but isn't gated on the results.


This brings up the larger issue -- PyPi is inherently different than these
efforts -- PyPi has always been about each package author maintaining their
own package -- Linux distros and conda-forge, and ??? all have a small set
of core contributions that do the package maintenance. This is a large
effort, and wold be insanely hard with the massive amount of stuff on
PyPi

In fact, I think the kinda-sort curation that comes from individual
communities is working remarkably well:

the scipy community
the django community
...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Mon, Nov 28, 2016 at 10:01 AM, Paul Moore  wrote:

> On 28 November 2016 at 17:53, Chris Barker  wrote:
> >> Why not just have a single pth file, maintained by the build
> >> tool, for all editable installs?
> >
> > shouldn't that be maintained by the install tool? i.e. pip -- the whole
> idea
> > is that the install tool is different than the built tool, yes? and
> adding a
> > package in editable mode is an installation job, not a build job.
> >
> > Also -- the idea here is that pip will know it's installed so it can
> > upgrade, de-install, etc, so it really is pip's job to maintain the
> > "editable_install.pth" file.
>
> Sorry - I was confusing "build tool" vs "install tool" here. Not
> intentionally, but the confusion is real.


it sure is!


> Setuptools is a build tool,
> and yet (currently) handles editable installs.


setuptools is a build tool, and an install tool, and a runtime resource
manager, and a package manger / install tool -- source of lots of
confusion, and what we are trying to get away from, yes? ;-)

So IMO, part of
> finalising editable install support would be thrashing out which
> aspects of the process are the responsibility of the build tool, and
> which the install tool. That's a non-trivial design job, so in the
> interests of keeping things moving, it seems to me that "defer a
> decision for now" remains the right decision here.


fair enough.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Thu, Nov 24, 2016 at 3:10 PM, Paul Moore  wrote:

> Honestly, I don't see what's so bad about pth files. They are a
> standard supported Python approach. Maybe setuptools' use of them is
> messy?


exactly.

The fact that setuptools over-uses (abuses?) pth files doesn't mean that
they can't be used reasonably.

and given the differences of filesystems, it's a not-too-bad way to
simulate symbolic links :-)

Why not just have a single pth file, maintained by the build
> tool, for all editable installs?


shouldn't that be maintained by the install tool? i.e. pip -- the whole
idea is that the install tool is different than the built tool, yes? and
adding a package in editable mode is an installation job, not a build job.

Also -- the idea here is that pip will know it's installed so it can
upgrade, de-install, etc, so it really is pip's job to maintain the
"editable_install.pth" file.

Most users would then only have one
> or two pth files.
>

let's us make sure we don't end up with the easy_install.pth nightmare!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Wed, Nov 23, 2016 at 4:32 PM, Nathaniel Smith  wrote:

> On Wed, Nov 23, 2016 at 3:14 PM, Chris Barker 
> wrote:
>


> > Please, please, let's figure SOMETHING our here - editable installs (or
> > "develop" installs) are a critical tool. Frankly, I don't know how anyone
> > can develop a package without them.
>
> Would a 'pip watch' command that watched your source tree and
> rebuilt/reinstalled it whenever you edited a file work for you?


probably, yes -- though that seems pretty inefficient -- I suppose ti could
be smart enough to only update the changed file(s), which wouldn't be too
bad.

as for rebuild -- setuptools' develop mode doesn't rebuild anyway -- you
have to rebuild by hand if you change anything that isn't pure python --
which frankly works fine, as extensions need o be rebuild anyhow.

What
> do you do for conda packages?


develop mode is for, well, developing -- so no need for a conda package,
you are working from source be definition, and conda is a packaging system,
not a build system (i.e. for python packages, conda build usually calls
setuptools (ideally via pip) to do the building anyway)

Does conda have an editable install
> equivalent?
>

despite what I said above, conda does have a develop command:

http://conda.pydata.org/docs/commands/build/conda-develop.html

It's needed because conda does some trickery with editing relative paths
for linking shared libs at install time. If you use setuptools' develop
mode with an extension module, it may find the wrong version of libs at run
time.

I honestly don't know how well it works -- I built my own kludge for my
stuff before it was ready to go:

https://github.com/NOAA-ORR-ERD/PyGnome/blob/master/py_gnome/re_link_for_anaconda.py

And this is a Mac only solution -- linking happens differently on Windows,
such that I think this is a non-issue, and not sure about Linux -- we are
deploying with conda on Linux, but I don't think anyone is developing on
Linux.


>  they make it harder
> to improve pip's infrastructure around upgrading, because upgrading
> requires pip to have a solid understanding of exactly what's
> installed. So it's absolutely possible to have some useful hack, like
> we do now, but useful hacks by their nature are hard to standardize:
>

True -- but I'd still rather a useful hack than nothing

standardization means "this is the final solution to this problem, it
> won't need further tweaking", and editable installs really feel like
> they might benefit from further tweaking. Or something.
>

maybe an officially marked as preliminary part of the standard?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Chris Barker]

On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver 
wrote:


> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
>

Please, please, let's figure SOMETHING our here - editable installs (or
"develop" installs) are a critical tool. Frankly, I don't know how anyone
can develop a package without them.

Back in the day I struggle mightily with kludging sys.path, and, relative
imports that never really worked right, and on and on -- it SUCKED.

Then I discovered setuptools develop mode -- yeah! IN fact, I don't think
I'd ever use setuptools at all if I didn't need it to get develop mode!

c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.
>

Not sure what that means -- does that mean that you couldn't get an
editable isntall with pip? but rather you would do:

setup.py develop   if you had setuptools as your build system, and

some_other_command  if you had some other build tool?

Not too bad, but why not have a standard way to invoke develop mode? If the
tool can support it, why not have a way for pip to tell an arbitrary build
system to "please install this package in develop mode"

On the other hand:

I've always thought we were moving toward proper separation of concerns, in
which case, pip should be focused on resolving dependencies and finding and
installing packages.

Should it even be possible to build and install a package from source with
pip?

But if it is, then it might as well support editable installs as well.

The complication I see here is that a tool can't know how to install in
editable mode unless it knows about the python environment it it running in
-- which is easy for a tool built with python, but a problem for a tool
written some other way.

However, I see from PEP 517:


The build backend object is expected to have attributes which provide some
or all of the following hooks. The commonconfig_settings argument is
described after the individual hooks:

def get_build_requires(config_settings):
...


So I guess we can count on a Python front end, at least, so 1(a) should be
doable.

In fact, is the user-install issue any different for editable installs than
regular ones?

-CHB













> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.
>
> Thanks,
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

[Chris Barker]

On Fri, Nov 4, 2016 at 11:29 PM, Nick Coghlan  wrote:

> If I understand correctly, conda-forge works on the same basic
> principle - reviewing the publishers before granting them publication
> access, rather than defending against arbitrarily malicious code at
> build time.
>

yup -- that's pretty much it. you need a conda-forge member to merge your
PR before you get a "feedstock" tied into the system.

I'm confused though -- IIUC, ANYONE can put something up on PyPi with
arbitrary code in it that will get run by someone when they do pip install
of it.

So how is allowing anyone to push something to PyPi that will run arbitrary
code on a CI server, that will push arbitrary code to PyPi that will then
get run by anyone that pip installs it?

Essentially, we have already said that there is no such thing as "trusting
PyPi" -- you need to trust each individual package. So how in any sort of
auto-build system going to change that??



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

Final note after a long thread:

Just like Nick pointed out in his original post (if I read it right) , the
pip vs the conda approach comes down to this:

Do you  want to  a system to manage the whole stack? or do you want a
system to manage Python packages?

Personally, I think that no matter how you slice it, someone needs to
manage the whole stack. I find it highly preferable to manage the whole
stack with one tool. And even more so to manage the stack on multiple
platforms with the same tool.

But others have reasons to manage part of the stack (python itself, for
instance) with a particular system, and thus only need a tool to manage the
python packages part.

Two different deployment use cases, that will probably always exist.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 3:02 PM, Paul Moore  wrote:

> It would buy *me* flexibility to use python.org build of Python, or my
> own builds. And not to have to wait for conda to release a build of a
> new version.


you are perfectly free to make your own conda package of python -- it's not
that hard.

In fact, that's one of the key benefits of conda's approach -- conda
manages python itself, so you can easily have a conda environment for the
latest development release of python.

of, for that matter, just build python from source inside an environment.

ANd if you want to build a custom python package, you don't even need your
own recipe:

https://github.com/conda-forge/python-feedstock/tree/master/recipe

and conda can provide all the dependencies for you too :-)


> Of course, I have specialised needs, so that's not an
> important consideration for the conda developers.
>

True -- though the particular need of using a custom-compiled python is
actually pretty well supported!

 Caveat: I've never tried this -- and since conda itself uses Python, you
may break conda if you install a python that can't run conda. But maybe not
-- conda could be smart enough to use its "root" install of python
regardless of what python you have in your environment. in fact, I think
that is the case. when I am in an activated conda environment, the "conda"
command is a link to the root conda command, so Im pretty sure it will use
the root python to run itself.




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 3:02 PM, Paul Moore  wrote:

>  it may be possible to define a
> >> standard approach that goes something along the lines of defining a
> >> standard location that (somehow) gets added to the load path, and
> >> interested parties provide DLLs for external dependencies, which the
> >> users can then manually place in those locations.
> >
> > that't pretty much what conda is :-)
>
> Well, it doesn't feel like that. Maybe we're not understanding each
> other. Am I able to take a non-conda installation of Python,


no -- not at all -- but there is no "system install" of Python on Windows
that your IT folks are telling you that you shoudl use.

(except maybe IBM, if I recall from a David Beazley talk)

So conda provides that as well.

But anyway, I meant _conceptually_ it's the same thing.

 use conda
> to install *just* a set of non-Python DLLs (libxml, libyaml, ...) and
> then use pip to install wheels for lxml, pyyaml, etc?


If you have a conda python already, then yes, you can install conda
packages of various libs, and then build pyton packages that depend on them.

And now that I think about it -- you could probably:

install conda (which WILL install python -- cond needs it itself...)

do some munging of environment variables.

Use another python install to build wheels, etc.

If you got your environment variables set up right -- then building from
anywhere should be able to find the conda stuff.

But I don't know that you'd get any of the advantages of conda environments
this way.

I'm still trying to figure out why you'd want to do that on Windows, though
-- why not let conda manage your python too?

that there currently isn't a way to *build* an lxml wheel that links
> to a conda-installed libxml, but that's not the point I'm making - if
> conda provided a way to manage external DLLs only, then it would be
> possible in theory to contribute a setup.py fix to a project like lxml
> that detected and linked to a conda-installed libxml. That single
> source could then be used to build *both* wheels and conda packages,
> avoiding the current situation where effort on getting lxml to build
> is duplicated, once for conda and once for wheel.


see above -- conda is putting dlls into a standard place -- no reason you
couldn't teach other systems to find them.

In fact, I did something like this on OS-X a long while back.

There was a project (still is!) called "Unix Compatibility Frameworks":

http://www.kyngchaos.com/software/frameworks

It has a bunch of the dependencies needed for various python packages I
wanted to support (matplotlib, PIL, gdal...).

So I installed that, then build bdist_mpgks against it (no wheel back
then!). I distributed those packages. In the install instructions, I told
folks to go install the "Unix Compatibility Frameworks", and then all my
packages would work.

Similar concept.

However, one "trick' here is multiple dependency versions. A given conda
environment can have only one version of a package. If you need different
version, you use different packages. but that would get ugly if you wanted
your external wheels to link to a particular package.

This all gets easier if you manage the whole stack with one tool -- that's
why conda is built that way.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Wed, Nov 2, 2016 at 5:02 PM, Matthew Brett 
wrote:

> Anaconda has an overwhelming advantage on Windows, in that Continuum
> can bear the licensing liabilities enforced by the Intel Fortran
> compiler, and we can not.


Technically, that's an advantage that a commercial distribution has --
really nothing to do with the conda technology per se. But yes, from a
practical perspective Continuum's support for conda an Anaconda is a major
bonus in many ways.

Note that continuum in now (I think) default ing to MKL also -- they seem
to have solved the res-distribution issues.

But the conda-forge project has its own builds of lots of stuff --
including the core scipy stack, based on OpenBLAS. There is a scipy build
there:

https://github.com/conda-forge/scipy-feedstock/tree/master/recipe

but alas, still stuck:

```
build:
# We lack openblas on Windows, and therefore can't build scipy there
either currently.
skip: true # [win or np!=111]
```

>   I'm sure you know, but the only
> practical open-source option is mingw-w64, that does not work with the
> Microsoft runtime used by Python 3.5 [1].


OT -- but is it stable for Python2.7 now?


> > But not pyHDF, netCDF5, gdal, shapely, ... (to name a few that I need to
> > work with). And these are ugly: which means very hard for end-users to
> > build, and very hard for people to package up into wheels (is it even
> > possible?)
>
> I'd be surprised if these packages were very hard to build OSX and
> Linux wheels for.  We're already building hard packages like Pillow
>

Pilllow is relatively easy :-) -- I was doing that years ago.


> and Matplotlib and h5py and pytables.


Do h5py and pytables share libhdf5 somehow? Or is the whole mess statically
linked into each?

  If you mean netCDF4 - there are
> already OSX and Windows wheels for that [2].
>

God bless Chris Gohlke  --- I have no idea how he does all that!

So take all the hassle of those -- and multiply by about ten to get what
the GIS stack needs: GDAL/OGR, Shapely, Fiona, etc

Which doesn't mean it couldn't be done -- just that it would be a pain, and
you'd end up with some pretty bloated wheels -- in the packages above, how
many copies of libpng will there be? how many of hdf5? proj.4? geos?

Maybe that simply doesn't matter -- computers have a lot of disk space and
memory these days.

But there is one more use-case -- folks that need to build their own stuff
against some of those libs (netcdf4 in my case). The nice thing about conda
is that it can provide the libs for me to use in my own custom built
projects, or for other folks to use to build conda packages with.

Maybe pip could be used to distribute those libs, too -- I know folks are
working on that.

Then there are non-python stuff that you need to work with that would be
nice to mange in environments, too -- R, MySQL, who knows?

As someone else on this thread noted -- it's jamming a square peg into a
round hole -- and why do that when there is a square hole you can use
instead?

At least that's what I decided.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 10:56 AM, Matthew Brett 
wrote:


> But - it would be a huge help if the PSF could help with funding to
> get mingw-w64 working.  This is the crucial blocker for progress on
> binary wheels on Windows.


for what it's worth, this is a blocker for many of for using Win64 at all!

As far as I know, conda has not solved this one -- maybe for Continuum's
paying customers?

(or for binaries that continuum builds, but not stuff I need to build
myself)

If it's just money we need, I'd hope we could scrape it together somehow!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 3:50 AM, Paul Moore  wrote:


> Even on the "hard" cases like Windows, it may be possible to define a
> standard approach that goes something along the lines of defining a
> standard location that (somehow) gets added to the load path, and
> interested parties provide DLLs for external dependencies, which the
> users can then manually place in those locations.


that't pretty much what conda is :-)

though it adds the ability to handle multiple environments, and tools so
you don't have to manually place anything.

It would probably be feasible to make a
conda-for-everything-but-python-itself. I'm just not sure that would buy
you much.

-CHB

Or there's the
> option that's been mentioned before, but never (to my knowledge)
> developed into a complete proposal, for packaging up external
> dependencies as wheels.
>

Folks were working on this at Pycon last spring -- any progress?



> In some ways, Windows is actually an *easier* platform in this regard,
> as it's much more consistent in what it does provide - the CRT, and
> nothing else, basically. So all of the rest of the "external
> dependencies" need to be shipped, which is a bad thing, but there's no
> combinatorial explosion of system dependencies to worry about, which
> is good


and pretty much what manylinux is about, yes?

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 3:39 AM, Nick Coghlan  wrote:

> I don't think there's much chance of any of this ever working on
> Windows - conda will rule there, and rightly so. Mac OS X seems likely
> to go the same way, although there's an outside chance brew may pick
> up some of the otherwise Linux-only capabilities if they prove
> successful.
>

this is a really good point then -- we're building  "platform independent
system that, oh actually is a "a few different Linux vendors" system. Which
is a perfectly worthy goal, but it was not at all the goal of conda.

I wasn't there, but I think conda started with the goal of supporting three
core platforms (that are pretty different) with the same UI, and as much of
the same code as possible. And if you want to do that -- it really makes a
lot of sense to control as much of the system as possible.


> What I'm mainly reacting to here is Chris's apparent position that he
> sees the entire effort as pointless.


I don't believe I actually ever said that -- though I can see why'd you'd
think so.

But for the record, I don't see it as pointless.

It's not pointless, just hard,
>

That I do see -- I see it as hard, so hard that it's likely to hit road
blocks, so we could make better progress with other solutions if that's
where people put their energies. And you said yourself earlier in this post
that:

 "I don't think there's much chance of any of this ever working on Windows"

One of the "points" for me it to support Windows, OS-X and Linux -- so in
that sense -- I guess it is pointless ;-)

By the way, I was quite disappointed when i discovered that conda has done
literally nothing to make building cross-platfrom -- I understand why, but
none the less, building can be a real pain.

But it's also nice that it has a made a nice clean distinction between
package management and building, a distinction that is all too blurred in
the distutils/setuptools/pip/wheel stack (which, I know, is slowly being
untangled..)

Not in the general case it won't, no, but it should be entirely
> feasible for distro-specific Linux wheels and manylinux1 wheels to be
> as user-friendly as conda in cases where folks on the distro side are
> willing to put in the effort to make the integration work properly.
>

I agree -- though I think the work up front by the distributors will be
more.


> It may also be feasible to define an ABI for "linuxconda" that's
> broader than the manylinux1 ABI, so folks can publish conda wheels
> direct to PyPI, and then pip could define a way for distros to
> indicate their ABI is "conda compatible" somehow.


Hmm -- now THAT's an interesting idea...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Thu, Nov 3, 2016 at 2:34 AM, Paul Moore  wrote:

> On 2 November 2016 at 23:08, Chris Barker  wrote:
> > After all, you can use pip from within a conda environment just fine :-)
>
> In my experience (some time ago) doing so ended up with the "tangled
> mess of multiple systems" you mentioned.


well, yes, my experience too, as a of two years ago,  -- which is why I'm
taken the time make conda packages of all the pure python libs I need (and
now that there is conda-forge, it's not a huge lift)


> Conda can't uninstall
> something I installed with pip,


sure can't! -- though that's too much of a barrier -- it'll give you an
error, and then you use pip to uninstall it.


> and I have no obvious way of
> introspecting "which installer did I use to install X?" And if I pip
> uninstall something installed by conda, I break things. (Apologies if
> I misrepresent, it was a while ago, but I know I ended up deciding
> that I needed to keep a log of every package I installed and what I
> installed it with if I wanted to keep things straight...)
>
> Has that improved?
>

yes, it has -- and lots of folks seem to have it work fine for them, though
I"d say in more of a one-way street approach:

- create a conda environment
- install all the conda packages you need
- install all the pip packages you need.

you're done. And most conda packages of python packages are built
"properly" with pip-compatible meta data, so a pip install won't get
confused and think it needs to install a dependency that is already there.

then you  can export your environment to a yaml file that keeps track of
which packages are conda, and which pip, and you (and your users) can
re-create that environment reliably -- even on a different platform.

I'm not trying to complain, I don't have any particular expectation
> that conda "has to" support using pip on a conda installation. But I
> do want to make sure we're clear on what works and what doesn't.


I'm not the expert here --  honestly do try avoid mixing pip and conda --
but support for it has gotten a lot better.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

Hey Matthew,

> [1] There seems to be some animosity among pip supporters and conda
> > supports, or at least a perception that there is.
>
> I don't know whether there is animosity, but there is certainly
> tension.  Speaking personally, I care a lot about having the option to
> prefer pip.


indeed -- and you have made herculean efforts to make that possible :-)


> There are others in the scientific community who feel we
> should standardize on conda.   I think this is the cause of Chris'
> frustration.  If we could all use conda instead of pip, then this
> would make it easier for him in terms of packaging, because he would
> not have to support pip  (Chris please correct me if I'm wrong).
>

yup -- nor would you :-)

My personal frustration comes with my history -- I've been in this
community for over 15 years -- and I spent a lot of effort back in the day
to make packages available for the Mac (before pypi, pip, before wheels,
...). And I found I was constantly re-figuring out how to build the
dependent libraries needed, and statically linking everything, etc... It
sucked. A lot of work, and not at all fun (for me, anyway -- maybe some
folks enjoy that sort of thing).

So I started a project to share that effort, and to build a bit of
infrastructure. I also started looking into how to handle dependent libs
with pip+wheel -- I got no support whatsoever for that. I got frustrated
'cause it was too hard, and I also felt like I was fighting the tools. I
did not get far.

Mathew ended up picking up that effort and really making it work, and had
gotten all the core SciPY stuff out there as binary wheels -- really great
stuff.

But then I discovered conda -- and while I was resistant at first, I found
that it was a much nicer environment to do what I needed to do. It started
not because of anything specific about conda, but because someone had
already built a bunch of the stuff I needed -- nice!

But the breaking point was when I needed to build a package of my own:
py_gd -- it required libgd, which required libpng, libjpeg, libtiff ---
some of those come out of the box on a Mac, it's all available from
homebrew, and all pretty common on Linux -- but I have users that are on
Windows, or on a Mac and can't grok homebrew or macports. And, frankly,
neither do all of my development team.

But guess what? libgd wasn't in conda, but everything else I needed was --
this is all pretty common stuff -- other people have solved the problem and
the system supports installing libs, so I can just use them. My work was SO
MUCH easier.  And especially my users have it so much easier, cause I can
just give them a conda package.

And while that particular example would have been solvable with binary
wheels, as things get more complicated, it gets hard or impossible to do.

So if I'm happy with conda -- why the frustration? Some is the history, but
also there are two things:

1) pip is considered "the way" to handle dependencies -- my users want to
use it, and they ask me for help using it, and I don't want to abandon my
users -- so more work for me.

2) I see people over and over again starting out with pip -- cause that's
what you do. Then hitting a  wall, then trying Enthought Canopy, then
trying Anaconda, then ending up with a tangled mess of multiple systems
where who knows what python "pip" is associated with. This is why "there is
only one way to do it" would be nice.


And I'm pretty sure that "wall" will always be there -- things have gotten
better with wheels -- between Matthew's efforts and manylinux, most
everybody can get the core SciPy stack with pip -- very nice!

But not pyHDF, netCDF5, gdal, shapely, ... (to name a few that I need to
work with). And these are ugly: which means very hard for end-users to
build, and very hard for people to package up into wheels (is it even
possible?)

And of course, all is not rosy with conda either -- the conda-forge effort
has made phenomenal progress, but it's really hard to manage that huge
stack of stuff (I'm using the time I'm writing this with to take a break
from conda-forge dependency hell ...). But in a way, I think we'd be better
off if there was more focus on conda-forge rather than the effort to
shoehorn pip into solving more of the dependency problem.

And the final frustration -- I think conda is still pretty misunderstood an
misrepresented as a solution only (or primarily) for "data scientists" or
people doing interactive data exploration, or "scientific programmers",
whereas it's actually a pretty good solution to a lot of people's problems.


> Although there are clear differences in the audience for pip and
> conda, there is also a very large overlap.   In practice the majority
> of users could reasonably choose one or the other as their starting
> point.


exactly.


>   Of course, one may come to dominate this choice over the
> other.  At the point where enough users become frustrated with the
> lack of pip wheels, conda will become the default.   If pip w

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Wed, Nov 2, 2016 at 11:31 AM, Donald Stufft  wrote:

> Sure. Do whatever you want, I don’t think anyone here thinks you
> absolutely must use pip. :) [1]
>

indeed -- and IIUC, part of the thrust of Nick's post was that different
package managers serve different use-cases -- so we do need more than one.


> My point is just that even if you narrow yourself down to CPython 3.6.0
> there are still variations that can cause problems so each project
> individually ends up needing to decide what they support. Often times
> “unsupported” variations will still work unless they go out of their way to
> break it, but not always.
>

yup -- when I say I don't support non-conda installs, it doesn't mean my
software wont work without it -- it means you're going to need to figure
out how to compile those ugly dependencies yourself :-)


> [1] There seems to be some animosity among pip supporters and conda
> supports, or at least a perception that there is.
>

for my part, I'd say "frustration" more than animosity. I see a lot of
folks struggling with tools that don't serve their needs, without awareness
that there are better options.

And some of those folks want me to support them -- and I do want to support
my users.

I’d just like to say that this isn’t really shared (to my knowledge) by the
> development teams of either project. I think everyone involved thinks folks
> should use whatever solution best allows them to solve whatever problem
> they are having.
>

I agree -- Python is a great community!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Wed, Nov 2, 2016 at 9:49 AM, Nick Coghlan  wrote:

> No, as the post was about the fundamental and irreconcilable
> differences in capabilities, not the incidental ones that can be
> solved if folks choose (or are paid) to put in the necessary design
> and development time.


It's your post, but the external dependency is hardly an incidental issue.


> The post isn't written for beginners deciding which tool to use, it's
> written for intermediate folks that have already chosen one or the
> other for their own needs, and are wondering why the other still
> exists (or, worse, are telling people that have chosen the other tool
> for good reasons that they're wrong, and should switch).


again -- those reasons are very much about external dependency management!

- you need a system for specifying environmental *constraints* (like
> dynamically linked C libraries and command line applications you
> invoke)
> - you need a system for asking the host environment if it can satisfy
> those constraints
>

and it it can't -- you're then done -- that's actually the easy part (and
happens already and build or run time, yes?):

I try to build libgdal, it'll fail if I don't have that huge pile of
dependencies installed.

I try to run a wheel someone else built -- it'll also fail.

It'd be better if this could be hashed out a compilation or linking error,
sure, but that's not goign to do a whole lot except make the error messages
nicer :-)


> dnf/yum, apt, brew, conda, et al all *work around* the current lack of
> such a system by asking humans (aka "downstream package maintainers")
> to supply the additional information by hand in a platform specific
> format.


if conda is a "platform", then yes. but in that sense pip is a platform,
too.

I'll beat this drum again -- if you want to extend pip to solve all (most)
of the problems conda solves, then you are re-inventing conda. If someone
doesn't like the conda design, and has better ideas, great, but only
re-invent the wheel if you really are going to make a better wheel.

However, I haven't thought about it carefully -- maybe it would be possible
to have a system than managed everything except python itself. But that
would be difficult, and I don't see the point, except to satisfy brain dead
IT security folks :-)


> > But the different audiences aren't "data science folks" vs "web
> developers"
> > -- the different audiences are determined by deployment needs, not
> domain.
>
> Deployment needs are strongly correlated with domain though, and
> there's a world of difference between the way folks do exploratory
> data analysis and the way production apps are managed in
> Heroku/OpenShift/Cloud Foundry/Lambda/etc.
>

sigh. not everyone that uses the complex scipy stack is doing "exploratory
data analysis" -- a lot of us are building production software, much of it
behind web services...

and that's what I mean by deployment needs.

 However, if you're specifically interested in web service
> development, then swapping in your own Python runtime rather than just
> using a PaaS provided one is really much lower level than most
> beginners are going to want to be worrying about these days - getting
> opinionated about that kind of thing comes later (if it happens at
> all).
>

it's not a matter of opinion, but needs -- if this "beginner" is doing
stuff only with pure-python packages, then great -- there are many easy
options. But if they need some other stuff - maybe this beginner needs to
work with scientific data sets.. then they're dead in the water with a
Platform that doesn't support what they need.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Wed, Nov 2, 2016 at 9:57 AM, Donald Stufft  wrote:

> > On Nov 2, 2016, at 12:49 PM, Nick Coghlan  wrote:
> >
> > I also mean 2.6 vs 2.7 vs 3.4 vs 3.5 vs 3.6, etc
>

Of course, but that has nothing to do with the package management system...


> There are still platform differences too, we regularly get bugs that only
> are exposed on Anaconda or on Ubuntu’s Python or on RHEL's Python or on
> Python.org’s OS X installers etc etc.
>

Yes, that is the challenge -- and one reason folks may be tempted to say we
should have ONE package manager -- to reduce the variation -- though I
don't think any of us think that's possible (and probably not desirable).

Basically every variation has a chance to introduce a bug of some kind, and
> if you’re around long enough and you’re used enough you’ll run into them on
> every system. As someone writing that code you have to decide where you
> draw the line for what you support or not (for instance, you may support
> Ubuntu/RHEL/Anaconda, but you may decide that any version of CPython
> running on HPUX is not supported).
>

or you may decide to ONLY support conda -- my use case is a big pile of
tangled dependencies (yes, lots o' scientific stuff) that is fairly easy to
manage in conda and freekin' nightmare without it.

Oh, and I'm putting it behind a web service, so I need to deploy on
locked-down servers

You CAN get all those dependencies working without conda, but it's a
serious pain. Almost impossible on Windows.

-CHB








-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Wed, Nov 2, 2016 at 7:32 AM, Nick Coghlan  wrote:

> > He mentioned that conda allows you to
> > manage the python run-time itself, which is in deed a nice feature, but
> > getting a python run-time as never been the hard part (maybe on Linux if
> you
> > want a different one than your system supplies).
>
> I didn't miss it accidentally, I left it out because it wasn't
> relevant to the post (which was about the ecosystem design direction,
>

I would argue that is is quite relevant -- talking about design decisions
without talking about the motivations and consequences of those decisions
is missing much of the point.

The issue here is that folks might read that post and think: "do I want to
manage my python install or not?" and think that's the only question they
need to ask to determine if pip or conda is right for them. But it's not at
all.

pip is about managing stuff WITHIN python -- that's why it can work with
any conforming python install. So that's the advantage of this design
decision. But it leads to a major limitation, EVEN if you only care about
python, because it can't (properly) manage the stuff outside of python that
python packages may need. I honestly have no idea if the original
motivation was specifically to have a system that could work with any
python install (maybe), but it certainly was designed specifically to
handle python packages, period.

conda started with the motivation of managing complex non-python
dependencies (initially, to support python) -- in order to do that
effectively, it has to operate outside the  python envelope, and and that
means that it really needs to manage python itself. I'm pretty sure that
managing python itself was a consequence of the design goals, not a primary
design goal.

not the end user features that make the desire to use pip
> incomprehensible to a lot of folks).
>
> Designing software assembly tools for interpreted software is a system
> integrator's game, and when people are writing Python code, there is
> one absolutely 100% unavoidable integration task: choosing which
> Python runtimes they're going to support.
>

hmm -- I don't think that's the code-writers job -- it's the deployers job.
Other than choosing which python *version* I want to use, I can happily
develop with system python and pip, and then deploy with conda -- or vice
versa. INdeed, I can develop on Windows and deploy on LInux, or

though if you meant pypy vs iron python vs cPython when you meant "runtime"
then yes, with the dependency issue, you really do need to make that choice
upfront.


>  conda requires that you use a
> runtime that it provides (similar to the way Linux distro packaging
> works), while pip really doesn't care where the runtime came from, as
> long as it can run pip properly.
>

yes indeed -- and I'm fully aware of the consequences of that -- I work in
a very locked down system -- our system security folks REALLY want us to
use system-supplied packages (i.e. the python supplied by the OS
distributor) where possible. But that doesn't mean that I should try to
manage my applications with pip -- because while that makes the python part
easier, it makes the dependencies a nightmare -- having to custom compile
all sorts of stuff -- much more work, and not any more satisfying to the
security folks.

So you really need to look at your whole system to determine what will best
work for your use case.


> Essentially all the other differences between the two systems stem
> from the different choice made in that core design decision, which is
> why the two aren't substitutes for each other and never will be - they
> solve different problems for different audiences.
>

Indeed they do.

But the different audiences aren't "data science folks" vs "web developers"
-- the different audiences are determined by deployment needs, not domain.

conda environments are kind like mini docker containers -- they really  can
make life easier for lots of use cases, if you can get your IT folks to
accept a "non standard" python.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Tue, Nov 1, 2016 at 5:19 AM, Nick Coghlan  wrote:

> It isn't that simple, as what you really want to automate is the
> *release process*, where you upload an sdist, and the wheels *just
> happen* for:
>
> - the Python versions you want to support (e.g 2.7, 3.4, 3.5)
> - the platforms you want to support (typically x86_64 for Windows, Mac
> OS X, manylinux1, and maybe 32 bit Windows as well)
>

indeed -- do take a look at conda-forge, they've got all that working.

and while it ultimately calls "conda build this_thing", it wouldn't be hard
to make that a call to build a wheel instead.


> Adding a new Python release or a new platform to the build
> configuration is currently an activity that requires per-project work
> when in theory a build service could just add it automatically based
> on when new releases happen.


hmm -- maybe we could leverage gitHub, like conda-forge does -- Warehouse
would actually push to a repo on gitHub that would then trigger the CI
builds -- though the sure seems cleaner for Warehouse to call teh CIs
directly.



>
>
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Chris Barker]

On Tue, Nov 1, 2016 at 2:50 AM, Nick Coghlan  wrote:

> > I wrote some lines, but I deleted my thoughts about the topic
> "Automating wheel creation", since
> > I am a afraid it could raise bad mood in this list again. That's not my
> goal.
>
> I currently see 3 main ways that could eventually happen:
>
> - the PSF sorts out its general sustainability concerns to the point
> where it believes it can credibly maintain such a service on the
> community's behalf
>

That would be great.


> - the conda-forge folks branch out into offering wheel building as
> well (so it becomes a matter of "publish your Python projects for the
> user level conda platform, get platform independent Python wheels as
> well")
>

I doubt that's going to happen, but conda-forge is an open source project
-- someone else could certainly make a friendly fork and make a wheel-forge.

- someone builds such a service independently of the current PyPI
> infrastructure team, and convinces package publishers to start using
> it
>

and starting with the nice tools conda-forge has built would be a good way
to get going.

There is an odd thing here, though -- conda-forge works because of two
things:

1) The gitHub workflow is helpful for developing and vetting recipes

2) gitHub already has arrangements with CI services -- so it's easy to
leverage them to do the building.

But in theory, the PSF could negotiate an arrangement with the CI services
to make it easy to call them from PyPi.

and (1) is really necessary because of two things -- the folks making teh
recipes generally are not the package maintainers. and, critical here --
there are a lot of issues with consistent dependency management -- which
his what conda can help with, and wheels just don't.

ANother key point about all of this:

Nick missed the key point about conda. He mentioned that conda allows you
to manage the python run-time itself, which is in deed a nice feature, but
getting a python run-time as never been the hard part (maybe on Linux if
you want a different one than your system supplies).

And it's not about needing a compiler -- wheels solve that -- and getting
the system compiler is not all that hard either -- on all three major OSs.

It's about the non-python dependencies -- all the various C libs, etc that
your python packages might need. Or, for that matter, about non-python
command line utilities, and who knows what else. Sure, this is a bigger
issue in scientific computing --  which is why conda was born there -- but
it's an issue in many other realms as well. For many years, a Web-dev I
worked with avoided using PIL because it was too much a pain to install --
that's pretty easy these days (thanks Pillow!) but it's not long before a
serious web service might need SOMETHING that's not pure python...

So -- an auto wheel building system would be nice, but until it can handle
non-python dependencies, it's going to hit a wall

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Chris Barker]

>
> I think Ryan may have typed that command at a Python prompt rather than
>> a system command prompt. Unfortunately the distinction often isn't clear
>> in examples, because the experienced developers writing the instructions
>> are used to guessing which commands are Python and which are system
>> commands.
>>
>
indeed -- not a rare error for newbies.


> One thing I'd quite like to see Python grow is a standard function to
>> install packages from inside Python.
>
>
indeed:

import pip

pip.install('package_name')

would make lots of sense, except:

1) It is going to have to be a different command than the shell command, so
the above confusion would still be an issue.

2) are there any technicals reasons this would fail -- i.e. sys.path would
have to be re-calculated at least -- any other re-initializaton that would
have to be done?

3) permissions could be an issue, you don't necessarily want python to be
running with the permissions to write to it's install dir

In short -- I like the idea, but I'm not sure it'll really help the newbie
confusion.

-CHB








> In R, the install.packages()
>> function is the default way to get third party packages, and I think
>> staying in one interactive prompt does make things easier for new users
>> to understand.
>>
>
> Ah, I suspect you're right.
>
> I have considered adding a "Python x.y Shell" start menu item that would
> configure PATH properly for commands like "python" and "pip".
> Instinctively, it seems this would be more useful than a direct shortcut to
> the executable, but at the same time I don't want to start competing with
> all the other app-specific shells out there.
>
> Cheers,
> Steve
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] When can we kill Python 2.6 support?

[Chris Barker]

> Looking at pure usage numbers for "modern" versions of pip (6, 7, and 8)
for
downloading from PyPI I see the usage is ~3% of downloads are via Python
2.6.

And a lot of those may be CI systems for packages that still support 2.6

deprecate away!

-CHB




On Sat, Sep 3, 2016 at 1:36 PM, Daniel Holth  wrote:

> It would be convenient to drop 2.6 in wheel too.
>
> On Sat, Sep 3, 2016, 14:14 Brett Cannon  wrote:
>
>> I think the fact that Python 2.6 is past EOL means it's definitely up for
>> consideration. As for the 3% usage, as a trite comparison that's the amount
>> of scientists who deny climate change. So IMO that suggests 2.6 is not used
>> enough to burden PyPA with the maintenance and those who still want to use
>> it can take over maintaining 2.6 compatibility.
>>
>>
>> On Fri, 2 Sep 2016 at 14:06 Donald Stufft  wrote:
>>
>>> The packaging tools generally support 2.6+ and 3.(2|3)+ and that's sort
>>> of been
>>> where they've been at for a while now. I would like to think about what
>>> we need
>>> to be to start considering Python 2.6 as "too old" to support. In pip we
>>> generally follow a usage based deprecation/removal of supported Pythons
>>> but we
>>> don't have any real guidelines for when something is at a low enough
>>> usage to
>>> consider it no longer supported and we instead just sort of wait until
>>> someone
>>> makes a case that it's "low enough".
>>>
>>> This issue tends to impact more than just pip, because once pip drops
>>> support
>>> for something people tend to start dropping it across the entire
>>> ecosystem and
>>> use pip's no longer supporting it as justification for doing so.
>>>
>>> I would like to take a look at Python 2.6 and try and figure out if
>>> we're at a
>>> point that we can deprecate and drop it, and if not what is such a point.
>>>
>>> Looking at pure usage numbers for "modern" versions of pip (6, 7, and 8)
>>> for
>>> downloading from PyPI I see the usage is ~3% of downloads are via Python
>>> 2.6.
>>> The only thing lower than Python 2.6 that is still supported is Python
>>> 3.3.
>>>
>>> Python 2.6 itself has been EOL since 2013-10-29 which is now just about
>>> 3 years
>>> ago. It's SSL module is not generally secure and requires the use of
>>> additional
>>> installed modules to get it to be so. I believe the only place to get a
>>> Python 2.6 that is "supported" is through the Enterprise-y Linux
>>> Distributions
>>> like RHEL/CentOS/etc.
>>>
>>> Do we think that a ~3% usage of Python 2.6 and being end-of-life'd for
>>> ~3 years
>>> is enough to start deprecating and dropping 2.6? If not what sort of
>>> threshold
>>> do we think is enough? It'd be nice to get the albatross of Python 2.6
>>> support
>>> off from around our necks but I'm not sure how others feel. Obviously
>>> all of
>>> the existing versions of all of the tooling will still be fully
>>> functional so
>>> Python 2.6 users will simply need to not upgrade their tooling to
>>> continue to
>>> work, *but* it also means that they will be left out of new packaging
>>> features
>>> (and likewise, people can't rely on them if they still wish to support
>>> 2.6).
>>>
>>> Thoughts?
>>>
>>> —
>>> Donald Stufft
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker - NOAA Federal]

>
> If we're going through all this trouble, isn't it better just to jump to .zip 
> files like every other distribution format in existence?

Yes. :-)

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What role to eggs still play?

[Chris Barker - NOAA Federal]

Thanks, I think I'm getting it.

About the toml file... the *-info metadata is a compiled artifact,
according to all the existing Python packages. Most sdists even have a
*.egg-info directory.


If it's a compiled artifact, then shouldn't it NOT be in a source dist?

It is inconvenient if you want to know the true dependencies without
running setup.py.


Isn't that what the toml file is for?

I think we are stuck with it, and it's not all bad - if there is some
useful metadata that doesn't affect the dependency resolver, and it is "too
static" or too cumbersome to write out by hand, there's a place to do that.


I'm trying to imagine what that info would be,  but I suppose there could
be meta data about a package that is generated at build time -- maybe info
about how it was built, for instance.

Hmm -- maybe you could put info in there about non-Python shared libs it's
linked to, for instance.

Eggs are the only way to add a zipped distribution to PYTHONPATH and have
> setuptools find the metadata
>

Can pip find it in a zipped package? Remember,  I don't care to support
setuptools only features anyway :-)

. Eggs are used by buildout, especially in the unzipped into a directory
> form
>
And they could still be used for their originally designed use as a plugin
> format.
>

If pkg_resources gets spun off, would it support that?

 *.egg-info or EGG-INFO is the predecessor of the *.dist-info format
> designed in PEP-376. You get *.egg-info whenever you install something
> with setuptools without going through bdist_wheel.
>

So setuptools_lite would write a  dist_info.

I take it pip looks for both?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] What role to eggs still play?

[Chris Barker]

Hi all,

starting a new thread, but this is related to the setuptols-_lite
discussion, and the legacy formats discussion. In another thread Donald had
a footnote:


[1] We can tackle egg at a later point, when setuptools either has support
> for Wheels
> or is less needed.


So I'm wondering -- does anything else (other than setuptools) depend on
eggs in any way? I know why I pip install stuff, I (always?) get egg-ish
things installed:

.egg-info

directories and all that. Honestly, I'm confused -- is that making an
actual egg? or is that name simply a legacy name for package meta data?

In any case, does pip, or anything else, require it?

For my part, I find it annoying, name aside -- it seems that all the
package meta-data should be there in the package source already
(pypacakge.toml?)

-Chris







-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

[Chris Barker]

On Thu, Aug 18, 2016 at 10:17 PM, Wes Turner  wrote:

> setuptools does all of these, yes? but think of these in terms of when
>> they come into play:
>>
>> build time:
>>- building a package
>>
>
> - building c extensions
> - building NumPy (fortran,)
>

exactly! which is WHY we want/need more flexibility in build tools --
distutils is a mess to extend to do other things. (though is does do a fine
job with pure C extensions, which is what it was designed for).



> install time: installing a package;
>>   - installing a package
>>   - dependency management
>>
> run time
>>   - run-time version management
>>   - plugin handling, etc.
>>
>
> - running tests
>   - (It's much faster/easier/consistent/reproducible to run tests when
> all of the dependencies are bundled into one executable ZIP (e.g. PEX))
>

hmm - it seems most of us use a test-running for this -- so I put that in a
separate category. But you need the package management tool to be able to
easily set up an environment for testing -- i.e install the package and all
its dependencies, then run the tests. I don't know anything about PEX, but
I'd really NOT want the test runner to rely on any particular installation
setup.

And, as it happend, test running is one thing setuptools does not
currently, do so, we have pytest and nose and ??? that actually ARE
separate pluggable test running systems.


>
>> so we have:
>>
>>  - The mythical build tool
>>
>
> - TBH, I don't know much about pyproject.toml
>   - "PEP 518 -- Specifying Minimum Build System Requirements for Python
> Projects"
>  https://www.python.org/dev/peps/pep-0518/
>

pyproject.toml is NOT a build tool at all -- but it is a step towards being
able to support other build and packaging tools.

so maybe version 2 of setuptools_lite should rely on pyproject.toml.


> And then there are a number of build tools which work with multiple
> languages (because just solving for Python doesn't get it):
>
> - Conda (Python)
>

conda IS NOT a build tool -- I was pretty disappointed when I realized
that, but what can you do? cross-platform, cross-language build tools are a
really hard problem -- no one wants to write another one :-). And this is a
good thing -- what we WANT is for the packaging tool and the build tools to
be separate.


>   - meta.yaml, build.sh, build.bat *
>

the meta.yaml provides all the meta-data (kind like a higher level
pyproject.toml, and then simply invokes the build scripts. but you need to
write those build scripts -- conda provides literally no help with that.
most of them simply invoke whatever build system the library you are trying
to [package already uses -- setuptools, make, cmake, 

However, since you brought up testing earlier -- conda does have a nice
system for testing -- it isn't a test runner, but it does provide isolated
environments (a job of the installer???) -- so conda-build can:

setup an environment for building
build the package
Then:

setup an environment for testing
installed the package that was just built
invoke a test-runner

This is REALLY nice -- you can have all your test code run with the built
package, with the environment that you have specified, on the platform you
have built for (on).

This has caught all sorts of stupid errors for me -- forgetting a
dependency, etc...

Whether that should be built into the packaging tool I suppose is
debateable. But maybe it's not -- conda-build is a separate tool -- it
relies on the conda system, but it does it's own thing. conda packages are
fairly simple archives -- it's very possible to create them with other
tools.

- Blaze (Java, Python?; Google) -> Open Sourced as Bazel
>
- Pants (Python; Twitter):
> - Buck (Java, Facebook):
> - Bazel (Java, Python; Google)
>

Maybe the alternative to setuptools_lite is to set up one of these other
tools to work well (easily) for building python packages. If it can do
everything that setuptools can do (that we want setuptools to do), and just
as easily, maybe that's the next step forward.

but if that isn't happening soon, then a setuptools_lite would be a useful
step forward.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

[Chris Barker]

On Wed, Aug 17, 2016 at 6:45 PM, Daniel Holth  wrote:

> And a while back I argued against setuptools-lite, because I thought it
> would not solve the poor extensibility problem that stems from its basic
> distutils derived design... which includes all the classes and subclasses
> that have to work together to do its thing. My own thinking is that
> setuptools is fine, except that when you need to extend it you have a big
> problem; so make it easy to replace.
>

I never had any fantasies that setuptool-lite would improve on this in a
way shape or form -- indeed, a better build system is likely to need to be
built from scratch, or on a totally different framework, anyway.


> Does pip monkey patch setuptools? I thought it just forced setuptools even
> if a package only asked for distutils.
>

good point -- not really a monkey patch, but it does force particular
configuration.


> So you are suggesting setuptools_lite would not honor the setup_requires =
> [] and test_requires = []
>

essentially, yes. Though It would be nice if that data could be available
for pip -- and I"m not sure how that is done at this point.


> and perhaps "setup.py install" would also throw an error?
>

Probably not -- but it could only install from local source -- like
distutils. Though if we could re-work it to only do develop mode and build
wheels, that may be good -- you'd need pip to install it.


> What else? I would suggest that to effectively argue the idea requires
> reading the source code and enumerating exactly which complexity goes away
> with the _lite version.
>

I don't know that reading the source is the issue -- I'm thinking entirely
in terms of stripping out the API, so there may be things that need to stay
in, but we don't expose them. though maybe even that's harder than I think.

But yes -- enumerating what would and wouldn't be supported would be a good
start. But here is the guideline:

It seems we have a (maybe vague) vision of what we want the tool ecosystem
to look like that involves separation of concerns, such that each component
is replaceable without having to change the others:

The concerns:
  - building a package
  - installing a package
  - dependency management
  - run-time version management
  - plugin handling, etc.
  - resource managment

setuptools does all of these, yes? but think of these in terms of when they
come into play:

build time:
   - building a package
install time: installing a package;
  - installing a package
  - dependency management
run time
  - run-time version management
  - plugin handling, etc.

so we have:

 - The mythical build tool
 - pip
 - maybe some of the setuptools spun off -- like pkg_resources

So why aren't we there yet?

One reason is that setuptools the only thing out there that does the
building and some of the run-time stuff. So people use it. And now you
can't replace any of it without making a mess.

However, pip_wheel have indeed addressed much of the install-time and
dependency management issues. So folks use them, and they work well.

So what problem do I want setuptool-lite to solve?

I actually have spent a lot of time using conda to do my package installing
and dependency management. So I want a way to build a python package that
JUST builds the package.

In practice, I've found (most) of the flags and work arounds to diable
etuptools "features", but really?

python setup.py install --single-version-externally-managed

Oh right, that's not enough, I need a record file:

setup.py install --single-version-externally-managed  --record record.txt

Then I want to install in develop mode:

python setup.py develop

OOPS! now it's easy installing a bunch of dependencies! arrgg!

OK, -- no-deps it is.

Google around, this kind of thing is a pain in a lot of people's butts
--even though it call all be worked around.

So my vision of setuptool-lite is that is simply does not do the things
that vision of a build tool should not do.

i.e.:

never easy-install anything!
don't try to resolve dependencies

others??

One "lite" version of setuptools that I personally think would be cool
> would be pkg_resources as a separate package on pypi. Setuptools would
> require it as a dependency.
>

Yes, I'd love to disable pkg_resources too, but couldn't until it was spun
off.

In fact, years ago, long before pip, setuptools drove me crazy with how it
combined run-time, build time and intall time stuff. A serious problem when
I wanted to bundle something up with py2app (or py2exe, or...).

So what would be accomplished?

package builders could simply put:

from setuptools_lite import setup

(etc.)

and then their user would never accidentally easy install anything, etc.
And it would serve as documentation / notification if a user was using a
what-should-be-deprecated feature.

By the way, this is totally acknowledged to be a hacky, temporary mess that
might only help a bit to get us closer from being locked into setuptools.

(though maybe it would be a place to work throug

Re: [Distutils] Future of setuptools and buildout

[Chris Barker]

>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
>
Yes, please!

I argued  a while back for a setuptools-lite -- it would do all the things
setuptools does that we think it _should_ do, and not do any of the others
-- like it wouldn't easy-install anything EVER.

most likely it would be a fork of the setuptools code with a bunch of stuff
disabled. Or maybe even a setting in setuptools itself:

import setuptools
setuptools.disable_legacy

from setuptools import setup, find_packages, ...

Why

Folks rely on the various "features" of setuptools, some important ones
like buildout. So it's going to be a long time before we can deprecate all
that in setuptools itself.

But we're trying for a future with better separation of concerns --
building, installing, packaging, run-time management.

But as it stands now, people kind of HAVE to use setuptools to get desired
build behaviour, but then their users an accidentally invoke features they
don't want -- to teh point where pip goes in and monkey patches the darn
thing.

IF there were a setuptools_lite, user could simply do:

import setuptools_lite as setuptools

and they'd instantly get an error if they were using depreciated features,
and their end users would never accidently easy install stuff :-)

Anyway, this seem like a path forward, without having to wait for the
future fabulous pluggable build system .

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker]

On Tue, Aug 16, 2016 at 9:10 AM, Donald Stufft  wrote:
>
> Ah, I knew I was forgetting something. I think we should hold off on
> preventing egg uploads until setuptools can download and install wheels.
>

Aren't we trying to get to the point where setuptools doesn't download and
install anything ?

Sigh.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker]

On Tue, Aug 16, 2016 at 8:51 AM, Brett Cannon  wrote:

> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists,
>

I do -- what advantage does tar.gz have over zip? Zip's always been
supported on any platform I've used...

But both are in python's stdlib, so I suppose everyone has it that has
Python.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] license for setuptools

[Chris Barker]

Sorry for the noise  -- I was way too late to this game -- blib in my
email, I guess.

-CHB


On Fri, Aug 12, 2016 at 10:01 AM, Chris Barker 
wrote:

> it looks like the OP is looking for the license to setuptools, not
> matplotlib.
>
> (MPL's licence file is here: https://github.com/
> matplotlib/matplotlib/blob/master/LICENSE/LICENSE)
>
> but setuptools does not appear to have one:
>
> https://github.com/pypa/setuptools
>
> and it is a pypa package, so this is kind the right list for it -- though
> I"d post an issue on gitHub.
>
> BTW, I notice that the Python Packaging User Guide doesn't recommend a
> LICENSE file -- it probably should.
>
> NOTE to the OP: almost every python package depends on setuptools, so this
> really has nothing to do with Matplotlib :-)
>
> -CHB
>
>
> On Thu, Aug 11, 2016 at 8:36 PM, Noah Kantrowitz 
> wrote:
>
>> Hi there, this list is for the discussion of Python's core packaging
>> tools like distutils. We have no control over the packages made or
>> distributed with them. You would have to contact the matplotlib authors,
>> not us.
>>
>> --Noah
>>
>> > On Aug 12, 2016, at 7:51 AM, Marinier, Claude 
>> wrote:
>> >
>> > Good afternoon (well it’s afternoon here in the EDT zone),
>> >
>> > I am in the process of requesting the installation of Python 3 with
>> matplotlib. The company needs to approve licenses but I cannot find the
>> license for setuptools. The description here says it uses an MIT license
>> but I cannot confirm this. On github, the file setup.py says the same thing.
>> >
>> > License :: OSI Approved :: MIT License
>> >
>> > Could the maintainer please add an explicit license file.
>> >
>> > I would really like to use matplotlib but will not get approval unless
>> we can confirm the license.
>> >
>> > Thank you.
>> >
>> > --
>> > Claude Marinier
>> >
>> >
>> > ___
>> > Distutils-SIG maillist  -  Distutils-SIG@python.org
>> > https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] license for setuptools

[Chris Barker]

it looks like the OP is looking for the license to setuptools, not
matplotlib.

(MPL's licence file is here:
https://github.com/matplotlib/matplotlib/blob/master/LICENSE/LICENSE)

but setuptools does not appear to have one:

https://github.com/pypa/setuptools

and it is a pypa package, so this is kind the right list for it -- though
I"d post an issue on gitHub.

BTW, I notice that the Python Packaging User Guide doesn't recommend a
LICENSE file -- it probably should.

NOTE to the OP: almost every python package depends on setuptools, so this
really has nothing to do with Matplotlib :-)

-CHB


On Thu, Aug 11, 2016 at 8:36 PM, Noah Kantrowitz 
wrote:

> Hi there, this list is for the discussion of Python's core packaging tools
> like distutils. We have no control over the packages made or distributed
> with them. You would have to contact the matplotlib authors, not us.
>
> --Noah
>
> > On Aug 12, 2016, at 7:51 AM, Marinier, Claude 
> wrote:
> >
> > Good afternoon (well it’s afternoon here in the EDT zone),
> >
> > I am in the process of requesting the installation of Python 3 with
> matplotlib. The company needs to approve licenses but I cannot find the
> license for setuptools. The description here says it uses an MIT license
> but I cannot confirm this. On github, the file setup.py says the same thing.
> >
> > License :: OSI Approved :: MIT License
> >
> > Could the maintainer please add an explicit license file.
> >
> > I would really like to use matplotlib but will not get approval unless
> we can confirm the license.
> >
> > Thank you.
> >
> > --
> > Claude Marinier
> >
> >
> > ___
> > Distutils-SIG maillist  -  Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip3 subversion user

[Chris Barker]

On Thu, Aug 4, 2016 at 11:36 AM, Weatherby,Gerard 
wrote:

> Apologies in advance if this is the wrong list for the question; if so,
> a pointer to the correct one would be appreciated.
>
> I'm trying to user the subversion option of pip3 to install a package; I
> can use
> sudo pip3 install svn+https://path_to_code , but this does not work
> because our subversion repository requires a subversion user name, and I
> don' t know to pass it through pip3 to subversion.
>
> Tried:
> sudo pip3 install --user gweatherby svn+https://path_to_code, but it
> still prompted for "root" password.
>

try:

sudo pip3 install  svn+https://gweatherby@path_to_code


I have no idea if it will work, but worth a shot


-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] bulk upload

[Chris Barker]

On Mon, Jul 25, 2016 at 8:55 AM, Robin Becker  wrote:

> In our private readonly pypi we have 93 releases. I don't think that
> burden should fall on pypi. However, it's not clear to me if I should push
> micro releases to pypi and then remove them when another release is made.
> Is there a way to remove a 'release' completely?


I'm pretty sure there is no way to remove a release (at least not
routinely). thi sis by design -- if someone has done something with that
particular release, we want it to be reproducible.

I see the point, but it's a little be too bad -- I know I've got some
releases up there that were replaced VERY soon due to a build error or some
carelessness on my part :-)

Apparently, disk space is cheap enough that PyPI doesn't need to worry
about it.

Are you running into any problems?

I did try to reduce my manylinux sizes by using a library of shared object
> codes (ie a .a built from the PIC compile objs), but I didn't seem able to
> make this work properly; the resulting .so seems to contain the whole
> library (freetype).


is this a problem other than file sizes? I think until / if Nathanial (or
someone :-) ) comes up with a standard way to make wheels of shared libs,
we'll simply have to live with large binaries.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Contributing money to package authors/maintainers via PyPI

[Chris Barker]

On Sat, Jul 23, 2016 at 12:22 PM, Nathaniel Smith  wrote:

> OTOH, if we give up on that part of the idea, then it becomes much easier
> :-). It'd be straightforward for PyPI to provide a "how to donate to this
> project" box on each project page, that has links to whatever donation
> processing service(s) the project prefers.
>
+1 on this one -- shift all the legal hassles to the projects, but provide
a tiny bit of infrastructure to make it easier for people to find.

note: for a higher level of support, the PSF _could_ follow the numfocus
approach:


NumFocus is a properly set-up non-profit that can act as a gateway for
particular projects, so that the individual projects don't need to set up
all that accounting and legal infrastructure:

http://www.numfocus.org/information-on-fiscal-sponsorship.html


However, there is still a pretty big barrier to entry to become a sponsored
organization, as there should be.

So I think it would be great if PyPi could simply put a donate button in
there, but not try to get into the money laundering business.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Chris Barker - NOAA Federal]

Getting to this thread late, but it didn't seem that was resolved in
the least, so I'll as my $0.02

> That overall got me thinking about namespace pollution in pip, that
> once something is pushed in, it's like to stay there forever.

This REALLY is a problem, and one that will only get worse. It would
be nice to work out an approach before it gets too bad.

> I'd rather see something similar to Linux distributions where there's
> a curated repository "core"

As pointed out, that's not going to happen.

> and a few semi-official, like "extra" and
> "community," and for some, "testing."

But namespaces like these could still be implemented without curation.

Right now, the barrier to entry to putting a package up on PyPI is
very, very low. There are a lot of 'toy', or 'experimental', or 'well
intentioned but abandoned' projects on there.

If there was a clear and obvious place for folks to put these packages
while they work out the kinks, and determine whether the package is
really going to fly, I think people would use it.

And we could have mostly-automatic policies about seemingly abandoned
projects as well -- moving them to the "unmaintained" channel, or

As for the 'deadman's switch' idea -- that's a great idea. Sure, there
are projects with no active maintainer that people rely on, but almost
be definition, if folks care about it, we'd be able to find someone
willing to put their finger on the dads man's button once a year

As for issues like setuptools and PIL, where there is a shift in
maintenance if a large, high profile package, we really need a
benevolent oligarchy for PyPA that can resolve these issues by hand.
As pointed out -- this really doesn't come up often.

However, in these discussions, I've observed a common theme: folks in
the community bring up issues about unmaintained packages, namespace
pollution, etc. the core PyPA folks respond with generally well
reasoned arguments why proposed solutions won't fly.

But it's totally unclear to me whether the core devs don't think these
are problems worth addressing, or think they can only be addresses
with major effort that no one has time for.

If the core devs think it's fine and dandy like it is, we can all stop
talking about it.

By the way, there is an experiment underway with a "curated" community
package repository for conda packages:

https://conda-forge.github.io

It's semi-curated, in the sense that only the core devs can approve a
package being added, but once added, anyone can maintain it.

It's going very well, but I'm not sure how it's going to scale. So
far, 900 or so packages, 80 or so maintainers. Which I think is very
impressive for a young system, but still a lot smaller than it could
be.

But I think PyPA should keep an eye on it -- I'm sure there will be
lessons learned.

-CHB






> A name foobar resolves to core/foobar- if that exists, and if
> not some subset of other repositories is used.
> This way, an outdated package can be moved to another repo without
> breaking install base.
>
> In fact, curation without namespaces will already be pretty good.
>
> d.
>
>> On 13 July 2016 at 19:24, Jim Fulton  wrote:
>>> On Tue, Jul 12, 2016 at 7:55 AM, Dima Tisnek  wrote:
>>> Hi all,
>>>
>>> Is anyone working on pruning old packages from pypi?
>>>
>>> I found something last updated in 2014, which, looking at the source
>>> appears half-done.
>>> Github link doesn't work any longer, no description, etc.
>>>
>>> I managed to find author's email address out of band, and he responded
>>> that he can't remember the password, yada yada.
>>>
>>> I wonder if some basic automation is possible here -- check if url's
>>> are reachable and if existing package satisfies basic requirements,
>>> failing that mark it as "possibly out of date"
>>
>> I'm curious why you view this as a problem that needs to be solved?
>>
>> - Do you want to take over the name yourself?
>>
>> - Are you afraid someone will stumble on this package and use it?
>>
>> Something else?
>>
>> Jim
>>
>> --
>> Jim Fulton
>> http://jimfulton.info
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to build python-packages depends on the output of other project

[Chris Barker]

First,

what you have is not all that inelegant -- it is the way to do it :-)

But there are a few options when you are wrapping a C/C++ lib for python:

Do you need to access that lib from other extensions or only from the one
extension? IF others, then you pretty much need to build a shared lib and
make sure that all your extension link to it. But if you only need to get
to it from one extension than there are three options:

1) don't compile a lib -- rather, build all your C/C++ code to the
extension itself. you can simply add the files to the "source" list -- for
a straightforward lib, this is the easiest way to go.

2) statically link -- build the lib as a static lib, and then link it in to
your extension. then there is no extra .so to keep track of and ship. at
least on *nix you can bypass teh linker by passing teh static lib in as
"extra_objects" -- I think. Something like that.

3) what you did -- build the .so and ship it with the extension.

HTH,

-Chris






On Thu, Jun 2, 2016 at 7:35 PM, Young Yang  wrote:

> Hi,
>
> My current solution is like this
>
> I get the source code of project A.   And use  `cmdclass={"install":
> my_install},`  in my setup function in setup.py.
>
> my_install is a subclass of `from setuptools.command.install import
> install`
> ```
> class my_install(install):
> def run(self):
> # DO something I want. Such as compiling the code of project A and
> copy the output of it (i.e. the .so file) to my binding folder
> install.run(self)
> ```
>
> At last I  add these options in my setup function in setup.py to include
> the shared library  in the install package.
> ```
>   package_dir={'my_binding_package': 'my_binding_folder'},
>   package_data={
>   'my_binding_package': ['Shared_lib.so'],
>   },
>   include_package_data=True,
> ```
>
> But I think there should be better ways to achieve these.
> Could anyone  give me any elegant examples to achieve the same goal?
>
> Thanks in advance
>
>
> On Thu, Jun 2, 2016 at 11:05 AM, Young Yang  wrote:
>
>> hi,
>>
>> I'm writing python-binding for project A.
>>
>> My python-binding depends on the compile output of project A(It is a .so
>> file), and the project A is not installed in the system(so we can't find
>> the .so files in the system libraries pathes)
>>
>> What's the elegant way to package my python-binding, so that I can
>> install everything by run `python setup.py` ?
>>
>> Any suggestions and comments will be appreciated :)
>>
>> --
>> Best wishes,
>> Young
>>
>
>
>
> --
> Best wishes,
> Young Yang
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] matrix: python_versions x supported_plattforms, cross-compiling vs VM

[Chris Barker]

Not that this isn't an issue, but:

On Fri, May 27, 2016 at 5:34 AM, Daniel Holth  wrote:

> So this was a problem with eggs too. Let's say ZODB 3.0.1 was just
> released. You are happily using 3.0.0, the next version is a minor upgrade,
> but there are no precompiled packages for 3.0.1, so your build breaks on
> Friday morning when you are trying to deploy.
>

If you are using pip, etc. to DEPLOY, they you'd better darn be using
explicit versions in your requirements.txt.

Just sayin'

Of course, you may not want to use explicit versions in development for CI
testing, and then you run into the same issue. But having a test fail on
commit is not nearly as big a deal

If you find yourself in that situation then you would appreciate an easy
> way to use the newest available binaries rather than the newest available
> source,
>

I think the trick here is that you need "binaries" for packages with C
extensions, but source is just fine for pure python.

So I kind of like the idea of making wheels the default for distributing on
PyPi always -- even for pure python modules. And wheels are trivial to
build from pure python packages -- so why not?

-CHB





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Chris Barker]

On Thu, May 26, 2016 at 11:22 PM, Wes Turner  wrote:

>
> "conda-forge  is a github organization
> containing repositories of conda recipes.
>

Just to be clear -- while conda-forge is about conda packages, a good deal
of the CI integration stuff would apply just as well to building wheels.

It would make a lot of sense to build off of what they've done.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Chris Barker]

On Wed, May 25, 2016 at 8:27 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> I think providing a self hostable build server which can be started with
> one command
> would be such a project.


The manylinux Docker container is heading in that direction already. I
suggest you consider helping out with that effort.

There is also the conda ecosystem, with the conda-forge project using
public CI systems to build packages:

https://conda-forge.github.io/

I imagine a similar system could be adopted for wheels

-CHB







>
> Regards,
>   Thomas Güttler
>
>
> --
> Thomas Guettler http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker - NOAA Federal]

​Not asking for any change but has anyone looked at libconfig
? ​It looks
quite interesting: simple grammar and nesting support. What do you think of
it


As pointed out, it's a C lib. But as we all like writing tools, it wouldn't
be very hard to write a Python parser for the format.

But it's a bit C-y for my taste, and yet another configure language? Really?

-CHB



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Chris Barker - NOAA Federal]

> When the upstream installation process is instead broken up into
> "build a binary artifact" and "install a binary artifact", that brings
> a few benefits:

Great -- thanks for the detailed explanation. Sounds like a good plan, then.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Fri, May 13, 2016 at 11:22 AM, Brett Cannon  wrote:


> No need to think; the decision is made and it's TOML. I know Chris doesn't
> mean to stir up trouble,
>

I got a bit out of sync with the conversation -- sorry for the noise.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Chris Barker]

On Fri, May 13, 2016 at 1:09 PM, Nathaniel Smith  wrote:

> But, the plan *is* to make wheels the standard way to build packages --
> that will be in the next pep :-). I'm not sure I'd call it "lock down",
> because there's nothing that will stop you running setup.py bdist_rpm or
> whatever. But our goal is to reach the point where package authors get a
> choice of what build system to use, and there's no guarantee that every
> build system will implement bdist_rpm.
>
> hmm -- this really feels like mingling packaging and building.

Does making sure everything builds a wheel help systems like rpm and the
like? Honestly I have no idea.

I do know that conda is very is very much designed to not care at all how a
package is build or installed, as long as it can be installed -- so if a
wheel is built and then that wheel is installed, that all the same to conda.

But is that the case for everything else?

I absolutely agree that we shouldn't expect a bdist_rpm and the like -- in
fact, those should all be deprecated.

but maybe a "install" that goes from source to installed package, without
passing through a wheel?

or maybe not -- I really don't know rpm or deb or anything else well enough
to know.

> So, the plan is to require all build systems to be able to output wheels,
> and then debian or conda-build or whoever will convert the wheel into
> whatever final package format they want.
>
easy for conda -- not sure about the others

hmm -- homebrew builds from source, so as long as you have a way to install
the wheel you built, it'll be fine (much like conda, but without ever
making a package)

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Chris Barker]

One other question:

Is it just examples or is "build" being defined as "build a wheel"?

i.e. there are times one might want to build a package without building a
wheel -- just it install it yourself, or to put it in another package
format -- conda, rpm, what have you.

In conda's case, building a wheel, and then installing it would work fine,
but I'm not sure we want to lock that down as the only way to build a
package.

Granted, if all it means is that someone will download an unnecessary
dependency, big deal.

I'm also a bit confused about whether we're trying to specify the
dependencies required simply to run the build tool itself, or the
dependencies required to actually do the build -- or the latter being saved
for another day?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Tue, May 10, 2016 at 1:54 AM, Paul Moore  wrote:

> I would love to use YAML. I really would. But for pip, we need a
> robust, easy to vendor Python implementation


conda has been using yaml forever (with pyyaml) , and whatever problem is
has (and there are many), I don't think I've ever seen a single issue cause
by yaml or pyyaml. Might be worth asking the conda devs.

I think we're freaking out way too much about what *could* go wrong.

Oh, and why not "JSON with comments and trailing commas" - it would be well
defined and easy to implement.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, best
option.

> After further consideration, and pytoml's author's comment about the spec
changing without a version increase, I think we might be better off rolling
our own.

>
> I like the general simplicity, and would stick with that, but I'd be a lot
> more comfortable if we had our spec that was more consistent.
>

If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was that
no other language has a lib to read it -- rolling out own does not solve
that!)

Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here",
...
}

or we could do "JSON with comments" -- not hard to write a tiny
pre-processor before passing it off to the json lib.

Anyway -- let's avoid the temptation to role your own everything, and use
something standard!

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sun, May 8, 2016 at 5:31 AM, Nick Coghlan  wrote:

> > any python developer is going to
> > run into these issues eventually...
>
> Aye, I know - conda was one of the systems I evaluated as a possible
> general purpose tool for user level package management in Fedora and
> derivatives (see
>
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement#Directions_to_be_Explored
> for details).
>
> Hence my comment about conda not currently solving *system integrator*
> problems in the general case - it's fine as a platform for running
> software *on top of* Fedora and for DIY integration within an
> organisation, but as a tool for helping to *build Fedora*, it turned
> out to introduce a whole slew of new problems


Sure -- I don't think it was ever intended to support rpm et. all -- it's
to be used INSTEAD of rpm et. all -- for the most part, rpm has solved the
problems that conda is trying to solve -- but only for rpm-based Linux
systems. And I'm going to guess that Continuum didn't want to:

build packages for rpm systems (which ones?)
build packages for deb-based systems (which ones?)
build packages for gentoo
build packages for arch..
.
build packages for homebrew

build packages for cygwin
build packages for Windows.. OOPS, there IS not package manager for
Windows!!

And, AFAICT, none of those package management systems support
"environments", either.

Clearly -- a tool like conda was required to meet Continuum's goals -- and
those goals are very much the same as PyPa's goals, actually. (except the
curated collection of packages part, but conda itself is not about the
Curation...)

However, the kinds of enhancements we're now considering upstream in
> pip should improve things for conda users as well - just as some folks
> in Fedora are exploring the feasibility of automatically rebuilding
> the whole of PyPI as RPMs


yes -- that's a good analogy -- for python packages, conda relies entirely
on distutils/setuptools/pip -- so yes, making those tools better and more
flexible is great.

but I'm still confused about what "the kinds of enhancements we're now
considering upstream in pip" are.

Here are a few:

More flexibility about the build system used
Easier to get metadata without actually initiating a build

These are great!

But I started this whole line of conversation because it seemed that there
was desire for:

Ability to isolate the build environment.
Ability to better handle/manage non-python dependencies

These are what I was referring to as mission-creep, and that overlap with
conda (and other tools).


> (twice, actually, anchored on
> /usr/bin/python and /usr/bin/python3), it should eventually be
> feasible to have the upstream->conda pipeline fully automated as well.


yeah -- there's been talk for ages of automatically building conda packages
(on the fly, maybe) from PyPi packages. But currently on conda-forge we've
decided to NOT try to do that -- it's turned out in practice that enough
pypi packages end up needing some hand-tweaking to build. So teh planned
workflow is now:

Auto-build a conda build script for a PyPi package
Test it
Tweak it as required
Add it to conda-forge.

Then -- *maybe* write a tool that auto-updates the PyPi based packages in a
chron job or whatever.

So not quite a automated conda-PyPi bridge, but not a bad start.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

>
> "pymeta" feels very "inessentially weird" to me [1].


yeah...

setup.toml

???

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Chris Barker]

On Sun, May 8, 2016 at 5:49 AM, Nick Coghlan  wrote:

> The reason I think "bootstrap" is a better name at this point


I *really* don't want to add to the bike-shedding of the name at this point
-- I really don't care.

I was just trying to see if I was misunderstanding something, as it didn't
seem to be bootstrapping anything to me, and I see I was kinda missing some
of the scope and complexity -- thanks.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker]

On Sat, May 7, 2016 at 6:04 PM, Brett Cannon  wrote:

For both options I hear "pick a new format", which suggests we might as
> well do it from the get-go for clear separation of the new stuff and just
> bite the bullet instead of simply postponing a decision; it isn't like our
> format options are going to significantly change between now and later in
> the year.
>

Agreed. However, in another thread, I understood you to say that ALL we are
talking about now is how to specify the build requirements. If that's the
case, then we might a well as well just go with setup.cfg.

However, I'd rather we were setting the stage for grater things -- in which
case, let's go with a new config file.

BTW, IIRC, there seemed to a consensus moving toward using a Python API,
rather than a command line API for the mythical pluggable build systems

In which case, we can require python, and could use python literals for
configuration. With the discussion of PyYaml, I"m thinking more and more
that something that can be parsed with only the stdlib is a good idea.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] who is BDFL for the boostrap/requires declaration? (was: moving things forward)

[Chris Barker]

On Sat, May 7, 2016 at 11:18 AM, Brett Cannon  wrote:

> What fields there will be and their semantics ...
>
>1. Format version (so just deciding on a name -- which also includes
>   whether it should be top-level or in a subsection -- and initial value)
>   2. The actual bootstrap field (so its name and what to do if a
>   dependency is already installed but at a version that doesn't match what
>   the bootstrap specification asks for)
>
> Am I missing anything?
>

So what is this new configuration file supposed to cover? All the package
meta-data? i.e. everything that would be needed by a package manager to
properly install the package?

or the build meta-data: everything needed by the build system to build the
package?

both in one file?

And am missing something? how is this about "bootstrapping" -- to me,
bootstrapping is when you need X to build X. Isn't this just regular old
configuration: you need x,y to build z?

-CHB







> And since I keep pushing on this I'm willing to be a co-author on any PEP
> if there's no hard deadline in getting the PEP written (i.e. I can help
> write the prose, but I don't have the time to do the coding as this will be
> the fourth PEP I have going in some state; got to catch up to Nick's 35
> PEPs somehow ;).
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Sat, May 7, 2016 at 6:17 AM, Greg Ewing 
wrote:

> Do you expect that
>
>> projects ... should (somehow) contain simplified instructions on how to
>> build the various C/Fortran extensions supplied in the bundle as
>> source code?
>>
>
> Essentially, yes. I'm not sure how achievable it would
> be, but ideally that's what I'd like.
>

I think we've all come to conclusion that that's simply not possible --
build configuration cannot be purely declarative in the general case -- if
you are building a package you are going to be running arbitrary code (and
really, if you're running a compiler, you're doing that anyway, so there
isn't an additional security issue here -- if you trust the code, you might
as well trust the build script)

> On a unix system, most of the time they would all be in

> well-known locations. If I install something in an unusual
> place or in an unusual way, I'm going to have to tell
> something about it anyway. I don't see how an executable
> setup file provided by the package author is going to
> magically figure out all the weird stuff I've done.
>

You can look at any number of packages -- essentially, they do what
configure scripts do with autotools -- various hacky ways to find the stuff
it's looking for.

and users really want this -- maybe on a "normal" *nix system there isn't
much to it, but on OS-X there sure is -- people may have hand installed the
dependencies, or they may have used fink, or macports, or homebrew, or...
and, except for the hand-install case, they may have no idea what the heck
they did and where it is.

I don't know if there are conventions for such things on
> Windows. I suspect not, in which case manual input is
> going to be needed one way or another.
>

Windows is even worse -- not only no conventions, but also no common
package manager, either (at least OS-X is limited to four :-) )

Not all of it, only the parts that strictly have to be
> performed as part of the build step of the install
> process, to use your terminology. That's a fairly
> restricted subset of the whole problem of compiling
> software.
>

I don't think it's possible (or desirable) to make a clear distinction
between "the build step of the install process" and building in general.

Could a purely declarative config file be flexible
> enough to handle this? I don't know. The distutils
> API is actually pretty declarative already if you
> use it in a straightforward way.
>

indeed -- and for the common cases, that works fine, but there's always
SOMETHING That some weird case is going to need.

You could, I suppose, separate out the configuration step from the build
step -- analogous to ./configure, vs make.

So the configure step would generate a purely declarative config file of
some sort, and then the build step would use that. In the simple case,
there might be no need for a configure step at all.

though I'm not sure even this is possible -- for instance, numpy used a
heavily enhanced distutils to do it's thing. Cython extends distutils to
understand the "build the C from the pyx" step, etc This is still used
decoratively, but it's third party code that is doing part of the build
step -- so in order to make the build system extendable, you need to it run
code

Anyway, I thought it was clear that we need to make the distinction between
building and installing/packaging, etc clear -- both form an API
perspective and a user perspective. So at this point, we need to establish
and API to the build system (probably compatible with what we have
(setup.py build) but we leave it up to the build system to figure out how
to do it's thing -- maybe someone will come up with a purely declarative
system -- who knows?

>  Running Pyrex to generate .c files

> from .pyx files is one that I've encountered.
> (I encouraged that one myself by including a distutils
> extension for it, which I later decided had been a
> mistake.)
>

I don't think it was a mistake :-) -- that's got to be done some time --
why add another layer???

That's nice, but it wouldn't help me when I encounter
> a package that *hadn't* been set up to use gregsbuild. :-(
>

sure -- but we can't have a one-build-system-to-rule-them-all until we can
first have a way to have ANY build system other than setuptools :-)

Key issue here:

Right now, in order to make things easier for users, and due to history,
the whole build/package/install processes are fully intermingled. I think
this is a BAD THING, for two reasons:

1) you can't replace any of the tools individually (except I suppose the
packaging part, as it's the last step)

2) user's don't know what's going on, or what's going to happen when they
try to "intsall" something:

It's great that you can "just run pip install" and it often "just works" --
but when it doesn't it's kind of ugly. And there are security concerns:
When a user runs:

pip install some_package

They don't know what's going to happen. pip goes and looks on PyPi to see
if it's there.

If it's there, it l

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan  wrote:

> On 7 May 2016 01:55, "Chris Barker"  wrote:
> > So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there)
>
> conda doesn't solve these problems either - it solves the *end user*
> problem for data analysts (install the Python library they want to use),
>
I really need to make this clear -- there is NOTHING "data analyst"
specific about these problems -- they do come up more in the computational
programming domain, but there are any number of other application that have
the same problems (pyQT has come up in this conversation, yes?) -- and
we're finding conda to be a good solution for our web development needs,
too -- a conda environment  is kinda like a lighter-weight, platform
independent docker container. And of course, there is more an more data
analysis going on behind web services these days too -- any python
developer is going to run into these issues eventually...

> by ignoring the system integrator one (interoperate with the system
> integrator's existing platform level package management systems, of which
> we all tend to have our own with no plans for consolidation)
>
> That's by design, though - conda was specifically created as a language
> independent cross-platform platform, not as a cross-platform plugin
> management system for Python runtimes.
>
exactly!

> For a long time I was convinced the existence of conda and Linux
> containers as end user tools meant we wouldn't ever need to solve these
> problems at the language runtime layer, but it's since become clear to me
> that there's significant value in solving these problems in a way that
> doesn't care about how you obtained your Python runtime.
>
yup -- that would be the scope creep I'm referring too :-)

But now I'm confused about what problems we're talking about. From above:

"""
interoperate with the system integrator's existing platform level package
management systems
"""

you mean rpm, .deb, homebrew, etc?  distutils did have (does it still)
bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt
to inter-operate with the system integrator's platform. But I think it
turns out that this is a hopeless task -- there are just too many different
systems to consider and maintain -- much better to let the external systems
handle that. And it got really ugly when you wanted to mingle virtualenv
with rpm, etc

The "solution" to that is to actually not do it -- use Docker or Conda if
you want an isolated environment.

What all this means to me is that  we really do need to keep the building
separate from the packaging -- so that the building tools can be used /
supported by the systems integrator. But people shouldn't be using rpm to
manage their system, and expect to install binary wheels...

The trick is that the pip/wheel system is really handy -- certainly for
Windows and OS-X that don't provide a standard system package manager. It
is nice to be able to "just pip install" various packages. But the problem
is that it is a half way solution -- it only works well for python-centered
packages -- pure python is trivial of course, but custom written C
extensions work great too -- but as soon as you get external dependencies,
it all goes to heck. We can improve that -- defining the manylinux platform
is a great step, but to really get there, there needs to be a way to handle
third party libs (which we could probably do with the existing tools by
establishing conventions), but then you get to non-python support tools:
Fortran compilers, other language run times, all sorts of stuff. Oh, and
then we want isolated environments, and they need to isolate these other
non-python tools, so 

I think it's WAY beyond the current PEP under discussion, but we really
should map out a "where we'd like to get" plan.

"significant value in solving these problems in a way that doesn't care
about how you obtained your Python runtime."

I don't get this -- I don't think it's possible to not care about the
python run-time -- we've only gotten this far because most folks have more
or less standardized on the python.org builds as the standard python build.
Even when you get your python elsewhere (Anaconda, for instance) they take
pains to keep it compatible.

I think we should count on python itself being managed by the same package
manager as the packages -- or at least conforming to a standard (which I
think we have with teh ABI tags, yes?)

BTW, I only recently learned that there are more "other" python builds out
there than I thought. Appar

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 9:39 AM, Donald Stufft  wrote:

> On May 6, 2016, at 11:54 AM, Chris Barker  wrote:
>
> So my point is about scope-creep -- if you want the PyPa tools to solve
> all these problems, then you are re-inventing conda -- better to simply
> adopt conda (or fork it and fix issues that I'm sure are there….)
>
>
> Adopting Conda is unlikely to ever happen for the defaulting tooling.
>

I didn't put the emphasis right in that sentence -- I was not actually
suggesting that conda be adopted -- what I was suggesting was the we DON'T
expand the scope of pip et al to be able to do everything that conda does.

That being said, I think there are some misunderstanding here that may be
relevant to this discussion:


> The problems that the default tooling attempt to solve are significantly
> harder than the problems that Conda attempts to solve
>

kinda-sorta -- conda is a packaging system. period, end of story (actually
it is an isolated environment system, too -- is that packaging? )-- it is
NOT a build system. Right now, you can invoke pip to:

* Find and install a binary package (wheel) -- this conda does
* Find, download, build and install install a source package -- nope, conda
doesn't do anything like that.
* Build and install a local package from source
* Install a local package from source in "develop mode" (editable mde) --
actually conda does have that -- though I'm not sure who well it works, of
it it's python specific.

But the fact that this all is (apparently) done by one tool is actually a
problem -- we have a tangled mess of binary and source and building and
installing, and it isn't obvious to the user what they are getting ) often
they don't care -- as long as it works :-) ) And, under the hood, it's all
driven by setuptools, which also has duplicate and slightly different
functionality that can be accidentally invoked (fun with easy_install!).

So I thought the goal now was to clean up an untangle this mess, yes?


and switching to it would be a regression.
>

only in the sense the switching to pip (and getting rid of setuptools)
would be a regression


> The primary benefit of Conda’s packages over Wheels are that they have a
> curated repository with people who are ensuring that things build correctly
> and since they don’t rely on authors to do it correctly, they don’t have to
> wait for, or convince authors to do this fresh new thing.
>

That simply is not true. Let's not mix social issues from technical ones.
Continuum set out to create a curated set of packages, yes -- but they
didn't put the enormous amount of work into conda for no reason -- they did
it because pip + wheel didn't meet their needs (and it doesn't meet a lot
of our needs, either). And, as I understand it, they came to the Python
community, and said something along the lines of "the packaging tooling
doesn't meet the needs of the scientific community, can we contribute
changes, improvements" -- and were told something along the lines of: "We
don't want the standard tooling to solve those problems -- go make what you
need, and then maybe we'll talk"

(I can dig out the message from Travis Oliphant about this if you like)

And they did go out and make something else, and they have released it as
an open source project, and a number of folks are adopting it to solve
their problems, too.


> The problem is, none of those benefits are something that would apply if
> we decided to throw away the 588,074 files that are currently able to be
> installed on PyPI.
>

conda aside, I think there is a real hang-up here. yes, of course you don't
want to throw away all that -- we absolutely need to maintain the current
packaging, building structure to support existing projects. But that
doesn't mean something new and better can't be built to be used in parallel
-- as folks move forward with pacakge development, they can adopt the new
system or they can keep using teh same old setup.py they have. MAybe
decades later something will get depreciated. But this approach is what got
us in the ugly mess to begin with.


Us deciding that Conda is the thing to use isn’t going to magically create
> an army of volunteers to go through and take all 80,000 packages on PyPI
> and ensure that we get a correctly compiled package on every platform for
> each version. If we could do that we could just convince everyone to go out
> and build binary packages, we could just do that with Wheels without
> requiring forklifting an entire ecosystem.
>

Again, conda packages and wheels are not the same thing -- there are
technical differences, and those differences really are why continuum uses
it for Anaconda, and why a lot of folks are adopting it for community led
efforts, like conda-forge. And there are a LOT of packages on conda-forge

Re: [Distutils] ez_setup.py can not get setuptools

[Chris Barker]

ez_setup.py is pretty darn old.

Any reason you can't:

python -m pip install setuptools

?

-CHB


On Fri, May 6, 2016 at 12:11 AM, Benedek Zoltan via Distutils-SIG <
distutils-sig@python.org> wrote:

> Hi,
>
> I don't know what happened recently. Usually I install setuptools by a
> script using the ez_setup.py script.
>
> Recently I get an error:
>
> Downloading
> https://pypi.python.org/packages/source/s/setuptools/setuptools-21.0.0.zip
> Traceback (most recent call last):
>   File "downloads/ez_setup.py", line 415, in 
> sys.exit(main())
>   File "downloads/ez_setup.py", line 411, in main
> archive = download_setuptools(**_download_args(options))
>   File "downloads/ez_setup.py", line 336, in download_setuptools
> downloader(url, saveto)
>   File "downloads/ez_setup.py", line 287, in download_file_insecure
> src = urlopen(url)
>   File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
> return opener.open(url, data, timeout)
>   File "/usr/lib/python3.4/urllib/request.py", line 469, in open
> response = meth(req, response)
>   File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
> 'http', request, response, code, msg, hdrs)
>   File "/usr/lib/python3.4/urllib/request.py", line 507, in error
> return self._call_chain(*args)
>   File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
> result = func(*args)
>   File "/usr/lib/python3.4/urllib/request.py", line 587, in
> http_error_default
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
> urllib.error.HTTPError: HTTP Error 404: Not Found
>
> For now I can copy the package from an old virtualenv, but I'd appreciate
> a better solution/advise.
>
> Thanks
> Zoltan
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:41 AM, Nick Coghlan  wrote:

> The "Python-with-imports" case is the status quo with setup.py, and we
> already know that's a pain because you need to set up an environment
> that already has the right dependencies installed to enable your
> module level imports in order to run the script and find out what
> dependencies you need to install to let you run the script in the
> first place.
>

good point -- this is really key.

The "Python-without-imports" approach would just be confusing -


I agree -- I never suggested that -- it's full python or fully declarative.


> So rather than saying "the bootstrapping dependency declaration file
> is Python-but-not-really", it's easier to say "it's an ini-file format
> that can be parsed with the configparser module" or "it's JSON" (I'm
> ruling out any options that don't have a stdlib parser in Python 2.7)
>

Last time, I promise :-)

"python literals" is perfectly well defined -- both by the language
reference, and by "can be parsed by ast.literal_eval"

and it addresses the limitations of JSON and is fully declarative.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Fri, May 6, 2016 at 5:55 AM, Nick Coghlan  wrote:

> > And maybe it's good to keep "new style" configuration clearly separate.
>
> Part of my motivation for suggesting re-using setup.cfg is the
> proliferation of packaging related config sprawl in project root
> directories - setup.py won't be going anywhere any time soon for most
> projects, some folks already have a setup.cfg (e.g. to specify
> universal wheel creation), and there's also MANIFEST.in to control
> sdist generation.


yeah -- ugly, but will  one more file make a difference? now that I think
about it -- IIUC the goals here, we want to separate packaging from
building -- I'd argue that setuup.cfg is about building -- we really should
keep the package configuration  separate. Some day, some how, we hoping to
have a new (or multiple build systems) they won't use setup.cfg to
configure the build. So we probably shouldn't marry ourselves to setup.cfg
for package configuration.

The other issue is social -- if we stick with setup.cfg, we are implying
that this is all about tweaking distutils/setuptools, not doing something
different -- seemingly keeping the mingling of building and packaging
forevermore

But this isn't that big a deal -- enough bike-shedding, time to make a
declaration.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 6:37 PM, Robert Collins 
wrote:

>
> Thats good. It occurs to me that scientific builds may be univerally
> broken because folk want to avoid easy-install and the high cost of
> double builds of things. E.g. adding bootstrap_requires will let folk
> fix things?
>

scientific builds are universally "broken" because their dependencies are
often non-python (and oftne complex and hard to build) external libs.

that is simply a problem that setuptools and pip were never designed to
address.


> But the main question is : why are these packages staying inaccurate?
> Even in the absence of a systematic solution I'd expect bug reports
> and pull requests to converge on good dependencies.
>

the problem is not that people haven't declared the proper dependencies in
their packages -- it's because it's impossible to declare the proper
dependencies in their packages...

The solution is to do one of three things:

1) punt -- and let folks use Canopy, or conda, or third party collections
of wheels like the Gohlke repository, or let folks build it themselves (a
lot of packages have all kinds of custom code to go look in macports, or
homebrew, or fink locations for libs on OS-X for instance.

2) build binary wheels that statically link in everything needed (or
include the dlls in the package)

3) make binary wheels out of C libs, so that other packages can depend on
them --  this is kind of a kludgy hack (abuse?) of pip/wheel but should be
do-able.

Some folks are making a valiant effort to do a combination of (2) and (3)
for Windows, OS-X, and now many linux -- but it is too bad that the tools
don't make it easy. And so far, all we really have are the core scipy stack
(and pyQT5???)

In short -- building fully supported binary packages is possible, but
requires a fair bit of expertise and a valiant effort on the part of
someone...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 4:32 PM, Nathaniel Smith  wrote:

> > You do know that we're on our way to re-writing conda, now, don't you?
> :-)
> >
> > I think we need to be careful of scope-creep...
>
> conda did not invent the idea of creating separate python environments
> for different tasks :-)


I'm not suggesting conda invented anything -- I'm suggesting it has
implemented many of the things being talked about here -- the truth is
conda was designed to solve exactly the problems that scientific python
packages had that pip+wheel+setuptools do not currently solve.

So my point is about scope-creep -- if you want the PyPa tools to solve all
these problems, then you are re-inventing conda -- better to simply adopt
conda (or fork it and fix issues that I'm sure are there)

On the other hand, improving the PyPa tools while maintaining their current
scope is a lovely idea -- but that means leaving isolation of build
environments etc, to external tools: Docker, VMs, conda..

Actually, conda has a good lesson here: conda is about packaging, NOT
building -- I was quite disappointed that conda provided NO support for
cross platform building at all -- but after using it a bit I realized that
that was a great decision -- if you want to support a wide variety of
packages, you really need to let the package authors use whatever the heck
build system they want -- you really don't want to have to implement (or
even debug) everyone else's builds.

And IIUC, that is the direction we are trying to go with pip now -- making
pip+wheel build-system independent -- good goal.

Which actually give me an idea: it seems we are very bogged down in
struggles with keeping backward compatibility, and coming up with a API for
an arbitrary build system. Maybe we can take a lesson from conda and
essentially punt:

conda works by reading a meta.yaml -- a purely declarative package
configuration. The actual building is done by calling a build script --
conda itself does not need to know or care what the heck is in that build
script -- all it needs to know is that it can be run. Why not keep the API
that simple for pip?

we could do total backward compatible, and simply say: pip will call
"python setup.py install".

And that's it, end of story -- individual packages could do ANYTHING in
that setup.py script -- current packages would be using setuptools, but pip
wouldn't need to care at all what was in there -- only know that it would
install the package.

And, I suppose, a setup.py bdist_wheel for building wheels, and setup.py
sdist for creating source distributions), though I'm not sure that those
need to be a standardized API -- or at least not the SAME standardized API.


-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Thu, May 5, 2016 at 10:45 PM, Greg Ewing 
wrote:


> Even if python is available, you might not want to run
> arbitrary code just to install a package.
>
> If a config file can contain executable code, then it
> can contain bugs. Debugging is something the developer of
> a package should have to do, not the user. In my experience,
> fixing someone else's buggy setup.py is about as much fun
> as pulling one's own teeth out with a blunt screwdriver.


well, in my experience, debugging my OWN buggy setup.py is about equally
pleasant!

But I think there is consensus here that build systems need to be
customisable -- which means arbitrary code may have to be run. So I don't
think this is going to help anyone avoiding dealing with buggy builds :-(

or are we talking about a config file that would be delivered with a binary
wheel? IN which case, yes, it should not have any executable code in it.

And anyway, it seems folks want to go with static config anyway -- clearly
separating configuration from customization, so I'll stop now.

I'd still like to be able to use Python literals, though :-)

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward

[Chris Barker]

On Thu, May 5, 2016 at 3:21 PM, Greg Ewing 
wrote:

> Chris Barker wrote:
>
> OK -- that's more or less my thought -- if it's  python that gets run,
>> then you've got your config generator built in -- why not?
>>
>
> The difference is that the config generator only gets run
> once, when the config info needs to get produced. If the
> config file itself is executable, it needs to be run each
> time the config info gets *used*


exactly -- maybe some run-time configuration specialization? though maybe
that's exactly what we want to avoid.



> -- which could be by a
> tool for which running arbitrary python code is awkward or
> undesirable.


This is python packaging, I'm still trying to see why you'd need to read
the config without python available.

But I think the key point is -- do we want to enforce that the config data
is static -- it WILL not change depending on when or how or where it is
read?

If so, then yeah, no runnable code.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

Last post! sorry to have not kept up last night

> to call the new feature setup_requires but some prefer to eliminate
> > uncertainty by calling it bootstrap_requires.
>
> The main advantage of a new feature name is that when someone searches
> the internet for "python bootstrap_requires", they won't find a
> decade+ worth of inapplicable documentation of the setuptools feature
> :)


new name for sure -- but I don't like bootstrap-requires -- is this
bootstrapping isn't it jsut building?

why not build_requires?

BTW, conda has:

requires:
   build:
   run:
   test:

(I think that's it)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 2:47 AM, Nathaniel Smith  wrote:

> > When you introduce isolation, the build will only have the standard
> > library + whatever is declared as a dep: and pyqt5 has no source on
> > PyPI.
>

so build isolation isolates you from arbitrary system libs? now you NEED to
ship all your non-trivial dependent libs. which I guess is the point, but
again, we're re-implementing conda here.

And the only time where it makes sense to force this build requirement is
when you are building binary wheels to put up on PyPi. So maybe part of
bdist_wheel??

OH -- and you need to make sure there is an isolated test environment too
-- which is a bit of pain, particularly when the tests aren't installed as
part of the package.

And, of course this means a version of virtualenv that isolates you from
system libs -- that might be quite hard to do, actually.

and again, conda :-)

-CHB





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] moving things forward (was: wheel including files it shouldn't)

[Chris Barker]

On Thu, May 5, 2016 at 2:10 AM, Nathaniel Smith  wrote:

> ...The main thing I want to point out though, is that all of these
> problems you're raising are complications caused entirely by wanting
> to avoid build isolation in the name of simplicity. If each package
> gets its own isolated build environment, then it can depend on
> whatever it wants without any danger of collision with the ambient
> environment.


You do know that we're on our way to re-writing conda, now, don't you? :-)

I think we need to be careful of scope-creep...

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig