[Distutils] Re: manylinux1 guidelines for zlib?

[Antoine Pitrou]

Updating on this. It seems the manylinux1 tooling may automatically link and 
bundle an old zlib version if you link dynamically against it. That can be 
counter-productive...

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UKG7EFGMHYIWU462L6BJGM2VOXLTJQ6M/

[Distutils] Re: Opinions on requiring younger glibc in manylinux1 wheel?

[Antoine Pitrou]

Nathaniel Smith wrote:
> It's naughty, you shouldn't do it, and the energy you put into making
> pseudo-manylinux1 wheels could probably be better put into making finishing
> up the manylinux2010 work – there's not that much to do.

Can you explain what's missing?

Paul Moore wrote:
> 1. It sounds like manylinux2010 may be what you want.

Definitely.

> 2. Maybe there's value in a tag that emphasises "current hardware" more than 
> backward compatibility?

I would say there's value in having two official manylinux flavors at once, for 
example manylinux2010 for maximum compatibility (it's already 8 years old as 
far as requirements go!) and manylinux2016 for recent systems compatibility. 
Later, manylinux2022 gets released as the "recent systems compatibility" 
standard and manylinux2016 becomes the "maximum compatibility" flavor.

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZZQP4KU7BLPFYRDLSFMWBDC4RUBJCZRH/

[Distutils] Re: Opinions on requiring younger glibc in manylinux1 wheel?

[Antoine Pitrou]

Paul Moore wrote:
> I'm not really familiar with manylinux1, but I'd be concerned if we
> started getting bug reports on pip because we installed a library that
> claimed to be manylinux1 and was failing because it wasn't. (And yes,
> packaging errors like this are a common source of pip bug reports).
> 
> It seems to me that it's defeating the purpose of having standards if
> people aren't willing to follow them...

I agree with that. OTOH it seems providing binary wheels is generally a strong
demand from the community. I would be fine with only providing conda packages 
myself.

By the way other packages are already doing worse:
https://github.com/tensorflow/tensorflow/issues/8802

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/S6WYTY7NAGODRAOEC6QOBNG5I7AOIVAT/

[Distutils] Re: Opinions on requiring younger glibc in manylinux1 wheel?

[Antoine Pitrou]

Trishank Kuppusamy wrote:
> We are looking for help to review manylinux2010, though:

Yes, but I'm not competent for that unfortunately. Sorry :-(

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3N2LKFFGWBSXNVNGOR32WNON6YMCDDPL/

[Distutils] Re: Opinions on requiring younger glibc in manylinux1 wheel?

[Antoine Pitrou]

Trishank Kuppusamy wrote:
> I think this will require updating the PEP, at the very least:

Sorry, there was a misunderstanding. Maybe I should have been clearer.
My question was about publishing deliberately incompatible manylinux1 wheels
(without changing the PEP).

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4JQDALH5AOKIKZYMKACM53EAM22NJJPV/

[Distutils] Opinions on requiring younger glibc in manylinux1 wheel?

[Antoine Pitrou]

Hi,

According to recent messages, it seems manylinux2010 won't be ready soon. 
However, the baseline software in manylinux1 is becoming very old. As an 
example, a popular C++ library (Abseil - https://abseil.io/) requires a more 
recent glibc (see 
https://github.com/abseil/abseil-cpp/commit/add89fd0e4bfd7d874bb55b67f4e13bf8beca762#diff-9f9c7fefa83b53e16cd568e31f1bfcb9R81).

What do you think of publishing manylinux1 wheels that would require a more 
recent glibc? This is being discussed currently for the pyarrow package.

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/M4MSVY5MPAPXFWHH4PBLE6PEBPOBIA44/

[Distutils] manylinux1 guidelines for zlib?

[Antoine Pitrou]

Hello,

Surprisingly, the manylinux1 spec doesn't seem to include the zlib in the list 
of known-to-be-available libraries (are there GNU/Linux systems out there 
without a zlib installed?).

Since I'm assuming several packages already had a need for that, is there a 
recommended way to link in the zlib as part of a manylinux1 wheel? Would you 
recommend static linking with a private version, or dynamic linking?

Regards

Antoine.
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/ZZG6GL3XTBLBJXSITYHEXMFKN43EREB7/

Re: [Distutils] What is the official position on distutils?

[Antoine Pitrou]

On Tue, 30 Aug 2016 14:44:10 +0100
Paul Moore  wrote:
> On 30 August 2016 at 13:38, Antoine Pitrou  wrote:
> > On Tue, 30 Aug 2016 13:34:22 +0100
> > Paul Moore  wrote:  
> >>
> >> From what I know, anyone who is actually hacking into the internal
> >> APIs to that level tends to use only distutils (probably because as
> >> you say, setuptools is even worse) and therefore falls into my
> >> category of "people affected by distutils changes who won't mind if a
> >> change is made to setuptools".  
> >
> > Except when, as you point out, pip injects setuptools nilly-willy when
> > running setup.py?  
> 
> I'm confused. People who hack into distutils internals can't cope with
> having setuptools injected (because their hacks conflict with
> setuptools' hacks).

I'm not sure what you mean by that.  I'm sure there are situations
where it works (I can't remember for sure whether it did the last time
I did such "hacking").

The thing is, more and more problem expect packages to install with
"pip install ." and they report bugs for that (the PyPA's official
discours is of course partly responsible for this, since it's
insisting so heavily on using pip and discouraging direct calls to
setup.py).

And the day people expect all libraries to be available as wheel files,
it will probably become worse, because I don't know how to produce
wheel files using plain distutils.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Antoine Pitrou]

On Tue, 30 Aug 2016 13:34:22 +0100
Paul Moore  wrote:
> 
> From what I know, anyone who is actually hacking into the internal
> APIs to that level tends to use only distutils (probably because as
> you say, setuptools is even worse) and therefore falls into my
> category of "people affected by distutils changes who won't mind if a
> change is made to setuptools".

Except when, as you point out, pip injects setuptools nilly-willy when
running setup.py?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Antoine Pitrou]

On Sun, 28 Aug 2016 23:36:43 +0100
Paul Moore  wrote:
> * as a result, there would have to be an *extremely* compelling reason
> to make a change in distutils rather than in setuptools - sufficient
> to justify the additional risk, the extra developer effort needed, and
> the fact that any such change is only going to benefit users of newer
> versions of Python

There is a problem with this: distutils and setuptools don't document
or specify their internal APIs.  Yet if you want to do something
advanced, you have to hook into the command classes and either call
their APIs or extend them.

Using distutils, this is already tedious, since you have to go read its
source code and try to figure out what happens (it's rarely obvious).

Using setuptools, this is even worse, though, because setuptools builds
on top of distutils by modifying the command classes, and so you have
to figure out the effect that monkeypatching has on the overall
behaviour (in addition to figure out what happens in distutils).


I don't know if that's bothersome enough to compound the fact that
changes made in setuptools can benefit all Python versions, but I think
it's useful to keep it in mind.


(that lack of programmability is really distutils' failure, of course,
and setuptools inherited that failure by inheriting distutils'
architecture.)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 527 - Removing Un(der)used file types/extensions on PyPI

[Antoine Pitrou]

On Thu, 25 Aug 2016 17:46:03 +0200
"M.-A. Lemburg"  wrote:

> +sources of truth for a single version. Having multiple sdists often
> times can
> +account for strange bugs that only expose themselves based on which
> sdist that
> +the person used.
> 
> You may not be aware, but developers that work on both Windows
> and Unix often have two sets of source code packages: one using
> Windows line ends, the other using Unix ones.

"often"? According to which sources? :-)

I do work on both Windows and Unix (though Linux is my primary
development platform), and I've never had a separate source code
package for Windows (with differing files, line endings or whatnot).

I also agree that having multiple sdists for a single release is a
recipe for weird issues.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Antoine Pitrou]

On Tue, 23 Aug 2016 10:36:35 +0100
Paul Moore  wrote:
> 
> I don't see any sign of *anyone* working on a curated distribution for
> Windows along the lines of Linux distros or Homebrew. (Unless you
> count cross-platform stacks like conda, which IMO are a different
> scenario than "system" Python installs).

Under Windows, there isn't much moral difference between a conda install
(really a Miniconda or Anaconda install) and a python.org Python
install.  You can even install Anaconda or Miniconda system-wide.

(Miniconda is a minimal install of Python + conda, while Anaconda is a
pretty large selection of packages in addition - though not the entire
official repo contents, and not counting community packages)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] setuptools 25.1.6 broke numpy.distutils on Windows

[Antoine Pitrou]

Hello,

Just a heads-up that latest setuptools is incompatible with
numpy.distutils on Windows:
https://github.com/pypa/setuptools/issues/728

The whole scientific community will suffer until this issue is solved
one way or the other (either on the setuptools side, or on the Numpy
side, or both).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Antoine Pitrou]

On Wed, 11 May 2016 18:32:30 +
Brett Cannon  wrote:
> >
> > The only open issue in the PEP at the moment is the bikeshedding topic of
> > what to name the sub-section containing the requirements: `[package.build]`
> > or `[package.build-system]` (we couldn't reach consensus among the three of
> > us on this). Otherwise the three of us are rather happy with the way the
> > PEP has turned out and look forward to this being the first step towards
> > allowing projects to customize their build process better!
> >
> 
> So far the votes for this open issue are:
> 
> build-system:
> 
>1. Nathaniel
>2. Ian
>3. Ethan
>4. Nick
>5. Paul
> 
> build:
> 
>1. Donald
>2. Daniel
> 
> build-configuration (write-in candidate):
> 
>1. Antoine

Ha :) I like "build" actually.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Antoine Pitrou]

On Thu, 12 May 2016 00:20:32 +1000
Nick Coghlan  wrote:
> 
> When I say "build system configuration" in the context of
> distutils/setuptools, I mean things like:
> 
> * MANIFEST.in
> * non-dependency related setup() arguments (packages, package_dir,
> py_modules, ext_modules, namespace_packages, entry_points,
> include_package_data, zip_safe, etc)
> * the Extension class and its parameters:
> https://docs.python.org/2/distutils/setupscript.html#describing-extension-modules
> 
> Those are the settings that actually tell the build system what to
> build and (in some cases) how to build it.

That's confusing :-) You should really call it "build configuration".

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for specifying build dependencies

[Antoine Pitrou]

On Wed, 11 May 2016 17:11:54 +1000
Nick Coghlan  wrote:
> 
> Take the default case: for a distutils/setuptools based project, the
> actual build settings are the arguments to setup() in setup.py, *not*
> these new settings in pyproject.toml. By contrast, the settings in
> [package.build-system] are the ones that tell pip and other installers
> what is needed to make "setup.py bdist_wheel" work (and, in the
> future, will tell them when to invoke something other than "setup.py
> bdist_wheel" to do the binary build)

Side question: if the build system needs configuring, is a
user-provided configuration file really the best place to do so?
People will end up having to copy and paste a bunch of configuration
directives that are not directly relevant to their own project (also
those directives will need maintaining as a build tool may evolve
its dependencies over time).

Alternative: have a single "build system" configuration directive:

  [package.build-system]

  tool = "foopackage:fooexe"

... which instructs the runner to install dependency "foopackage", and
then invoking "fooexe" with a single well-known option (e.g.
"--pybuild-bootstrap-config") produces a JSON file on stdout describing
how to invoke precisely the tool for each build scenario (sdist, wheel,
etc.).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:55:38 -0400
Donald Stufft  wrote:
> 
> I think TOML is more usable than ConfigParser and in particular I think that
> the adhoc post processing step makes ConfigParser inherently less usable
> because it forces a special syntax that is specific to this one file. It also
> means that there's no "right" answer for when you have two different
> implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users
that it's worthwhile bothering them with a syntax change that will
require (at one point or another) migrating existing files. TOML doesn't
seem that compelling to me in that regard (quite less than YAML, and I'm
not a YAML fan).

(as an aside, if there's the question of forking an existing parser
implementation for better vendorability, forking a YAML parser may be
more useful to third-party folks than forking a TOML parser :-))

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:24:10 -0400
Donald Stufft  wrote:
> 
> TOML is infinitely better at nested structured that ConfigParser, given that
> TOML actually *supports* nested structures beyond a level of 1. The only way
> to get anything like:
> 
> [package.build]
> dependencies = ["setuptools", "wheel"]
> 
> In ConfigParser is to add post-processing to the values, which then you're no
> longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off
> code you wrote to do post processing" file.

The post-processing doesn't seem difficult enough to make any fuss
about it, IMHO.  The most important concern here should be how usable
the format is for end users, not whether implementations need 20
additional lines of code to work with it.

(also, what is wrong with providing a pypa-specific library for parsing
required configuration? are distlib / distil / pkgutil / the
distutils-competitor-du-jour still alive?)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Antoine Pitrou]

On Tue, 10 May 2016 10:38:51 +0300
Alex Grönholm  wrote:
> TOML isn't much better than ConfigParser in terms of representing nested 
> structures.

Indeed, that seems to be a strong point against TOML.  If we don't care
about nested structures that much, then ConfigParser should be more or
less ok...

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] For maximum performance, Python packages are best installed as zip files.

[Antoine Pitrou]

On Mon, 11 Apr 2016 07:40:05 -0400
Donald Stufft  wrote:
> 
> > On Apr 11, 2016, at 7:36 AM, Antoine Pitrou  wrote:
> > 
> > There is a certain number of disk accesses
> > that is necessary to get the required metadata in a correct way.
> 
> Right, but a SSD will perform those disk accesses quicker. Obviously doing 
> less of them is better than doing more of them faster, but doing less of them 
> faster seems to me to be better than doing less of them slower?

Yes, that's true.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] For maximum performance, Python packages are best installed as zip files.

[Antoine Pitrou]

On Mon, 11 Apr 2016 07:33:05 -0400
Donald Stufft  wrote:
> 
> > On Apr 11, 2016, at 7:23 AM, Antoine Pitrou  wrote:
> > 
> > On Mon, 11 Apr 2016 07:08:19 -0400
> > Donald Stufft  wrote:
> >> 
> >> I’m not sure if that is still the case with modern SSDs, but I think the 
> >> idea is that by putting everything inside of zip files you reduce the 
> >> number of stat calls that Python needs to do (they flip side of this is 
> >> that pkg_resources is incredibly slow because it needs to issue a ton of 
> >> stat calls on import).
> > 
> > I don't think SSDs have anything to do in it since the kernel should
> > cache directory contents, rather it's the number of system calls issued.
> > But as Paul says, in Python 3 importlib got a lot of optimization work
> > on this front, so this advice probably doesn't apply anymore.
> > 
> 
> Surely SSDs are faster at returning the data (metadata or actual data) than 
> spinning rust and thus would speed up importing on their own. Sure you might 
> have that data cached in memory if you’ve recently accessed it, but you just 
> as easily might not have that data cached and the OS might need to hit the 
> disk to find out.

That doesn't really follow. There is a certain number of disk accesses
that is necessary to get the required metadata in a correct way. That
number of accesses hasn't changed between Python 2 and Python 3. What
has changed is the number of spurious stat() calls asking for
information that can be cached. In Python 2, the OS would have done the
caching (sparing disk accesses but not system calls). On Python 3, the
caching is done inside importlib, which spares system calls in addition
to disk accesses.

(yes, this is a bit simplified because of NFS and friends ;-))

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] For maximum performance, Python packages are best installed as zip files.

[Antoine Pitrou]

On Mon, 11 Apr 2016 07:08:19 -0400
Donald Stufft  wrote:
> 
> I’m not sure if that is still the case with modern SSDs, but I think the idea 
> is that by putting everything inside of zip files you reduce the number of 
> stat calls that Python needs to do (they flip side of this is that 
> pkg_resources is incredibly slow because it needs to issue a ton of stat 
> calls on import).

I don't think SSDs have anything to do in it since the kernel should
cache directory contents, rather it's the number of system calls issued.
But as Paul says, in Python 3 importlib got a lot of optimization work
on this front, so this advice probably doesn't apply anymore.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] abstract build system approaches redux

[Antoine Pitrou]

On Thu, 3 Mar 2016 08:44:56 -0500
Donald Stufft  wrote:
> 
> I'd like to push back against this, speaking as someone who was originally pro
> CLI:
> 
> I think that a Python API is actually better for one reason: introspection.
> I cannot think of a particularly great way to have a CLI based build tool
> *evolve* with new APIs that are not user facing without requiring end users
> to do something like mark "ok now my thing is X compatible" or without
> inventing some sort of protocol negotiation phase.

I'll add that some build systems may have a non-trivial startup cost
(for example conda-build sets up an isolated environment with
well-defined binaries in it), therefore issuing several CLI commands
can be significantly more costly (and/or difficult to optimize for) than
issuing several API calls from the same single process invocation.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Antoine Pitrou]

On Wed, 17 Feb 2016 05:12:48 -0500
"Eric V. Smith"  wrote:

> On 2/17/2016 3:55 AM, Antoine Pitrou wrote:
> > On Tue, 16 Feb 2016 16:10:34 -0800
> > Glyph Lefkowitz  wrote:
> >>
> >> I am 100% on board with telling people "don't use `sudo pip install´".  
> >> Frankly I have been telling the pip developers to just break this for 
> >> years (see https://pip2014.com, which, much to my chagrin, still exists); 
> >> `sudo pip install´ should just exit immediately with an error; to the 
> >> extent that packagers need it, the only invocation that should work should 
> >> be `sudo pip install --i-am-building-an-operating-system´.
> > 
> > This is frankly ridiculous. The problem is not the use of "sudo" or the
> > invocation under root, it's to install into a system Python. So the
> > solution should be to flag the system Python as not suitable for using
> > pip into, not to forbid using pip under root.
> 
> I agree that there are uses for running pip as root (I do so myself).
> It's installing into the system Python that needs to be strongly
> discouraged, if not outright prevented. I'm not sure we have a good way
> of identifying the system Python, but that's another issue.

I think it would be reasonable to let vendors do so by placing a
specific file into the "site-packages" (or perhaps even a pip config
file if we want to put more information in it).  Then upstream Python
doesn't need to have a say in it.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Antoine Pitrou]

On Tue, 16 Feb 2016 16:10:34 -0800
Glyph Lefkowitz  wrote:
> 
> I am 100% on board with telling people "don't use `sudo pip install´".  
> Frankly I have been telling the pip developers to just break this for years 
> (see https://pip2014.com, which, much to my chagrin, still exists); `sudo pip 
> install´ should just exit immediately with an error; to the extent that 
> packagers need it, the only invocation that should work should be `sudo pip 
> install --i-am-building-an-operating-system´.

This is frankly ridiculous. The problem is not the use of "sudo" or the
invocation under root, it's to install into a system Python. So the
solution should be to flag the system Python as not suitable for using
pip into, not to forbid using pip under root.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Does anyone understand what's going on with libpython on Linux?

[Antoine Pitrou]

On Sun, 7 Feb 2016 00:25:57 -0800
"Robert T. McGibbon"  wrote:
> > What are Debian/Ubuntu doing in distutils so that extensions don't link
> to libpython by default?
> 
> I don't know exactly, but one way to reproduce this is simply to build the
> interpreter without `--enable-shared`.

See https://bugs.python.org/issue21536. It would be nice if you could
lobby for this issue to be resolved... (though that would only be for
3.6, presumably)

> I don't know that their reasons are, but I presume that the Debian
> maintainers have a well-considered reason for this design.

Actually, shared library builds can be noticeably slower. I did
measurements some time ago, and the results are:
- shared builds are 5-10% slower on x86
- they can be up to 30% slower on some ARM CPUs!
(this is on pystone which is a very crude benchmark, but in this case I
think the pattern is more general, since any function call internal to
Python is affected by the difference in code generation: shared library
builds add an indirection overhead when resolving non-static symbols)

Note btw. that Anaconda builds are also shared library builds.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Status update on the NumPy & SciPy vs SSE problem?

[Antoine Pitrou]

On Fri, 5 Feb 2016 21:46:54 +1000
Nick Coghlan  wrote:
> 
> Thanks for the replies, folks!
> 
> Checking I've understood the respective updates correctly:
> 
> - x86_64 implies SSE2 capability
> - most i686 machines still in use are also SSE2 capable
> - Accelerate provides native BLAS/LAPACK APIs for Mac OS X
> - (ATLAS SSE2 or OpenBLAS) + manylinux should handle Linux
> - (ATLAS SSE2 or OpenBLAS) + mingwpy.github.io should handle Windows
> - Numba can optimise at runtime to use newer instructions when available
> 
> The choice between an SSE2 build of ATLAS and OpenBLAS as the default
> BLAS/LAPACK implementation doesn't appear to have been made yet, but
> also shouldn't significantly impact the user experience of the
> resulting wheels.

I'm not sure that's what you're implying, but the choice of a specific
BLAS or LAPACK implementation needn't (and shouldn't) be part of
manylinux, it's just a choice left to the packager.  Bottom line is
that the BLAS/LAPACK implementation comes linked into the specific
package (or as a separate package dependency, up to the packager's
preference).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Status update on the NumPy & SciPy vs SSE problem?

[Antoine Pitrou]

On Thu, 4 Feb 2016 21:22:32 +1000
Nick Coghlan  wrote:
> 
> I figured that was independent of the manylinux PEP (since it affects
> Windows as well), but I'm also curious as to the current status (I
> found a couple of apparently relevant threads on the NumPy list, but
> figured it made more sense to just ask for an update rather than
> trusting my Google-fu)

While I'm not a Numpy maintainer, I don't think you can go much further
than SSE2 (which is standard under the x86-64 definition).

One factor is support by the kernel. The CentOS 5 kernel doesn't
seem to support AVX, so you can't use AVX there even if your processor
supports it (as the registers aren't preserved accross context
switches). And one design point of manylinux is to support old Linux
setups... (*)

There are intermediate ISA additions between SSE2 and AVX (additions
that don't require OS support), but I'm not sure they help much on
compiler-vectorized code as opposed to hand-written assembly.  Numpy's
pre-compiled loops are typically quite straightforward as far as I've
seen.

One mitigation is to delegate some operations to an optimized library
implementing the appropriate runtime switches: for example linear
algebra is delegated by Numpy and Scipy to optimized BLAS and LINPACK
libraries (which exist in various implementations such as OpenBLAS or
Intel's MKL).

(*) (this is an issue a JIT compiler helps circumvent: it generates
optimal code for the current CPU ;-))

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Non English Speaking Users of PyPI - I need Help!

[Antoine Pitrou]

On Tue, 26 Jan 2016 14:10:28 -0500
Donald Stufft  wrote:
> 
> Hopefully this was useful information!

It was certainly comprehensive. Thank you :-)

About gettext's shortcomings... I think it's so widely used that it's
probably possible to work around them whenever they appear. I'm not
saying the gettext API is pretty, though (especially the clumsy catalog
management commands).

By the way, the Babel library on PyPI may at least render some of that
stuff a bit easier to deal with.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Non English Speaking Users of PyPI - I need Help!

[Antoine Pitrou]

On Tue, 26 Jan 2016 12:16:16 -0500
Donald Stufft  wrote:
> 
> As many of you are aware there has been an effort to replace the current PyPI 
> with a new, improved PyPI. This project has been codenamed Warehouse and has 
> been progressing nicely. However we’ve run into a bit of an issue when 
> deciding what to support that we’re not feeling super qualified to make an 
> informed decision on.
> 
> The new PyPI is going to support translated content (for the UI elements, not 
> for what people upload to there), although we will not launch with any 
> translations actually added besides English. Currently the translation engine 
> we’re using (l20n.js) does not support anything but “Evergreen” browsers 
> (browsers that constantly and automatically update) which means we don’t have 
> support for older versions of IE. My question to anyone who is, or is 
> familiar with places where English isn’t the native language, how big of a 
> deal is this if we only support newer browsers for translations?
> 
> If you can weigh in on the issue for this 
> (https://github.com/pypa/warehouse/issues/881) that would be great! If you 
> know someone who might have a good insight, please pass this along to them as 
> well.

Not answering your question, but needing Javascript on the
client to support L10n sounds like a weird decision (although Mozilla
seems to be pushing this... how surprising).  Every bit of client-side
Javascript tends to make Web pages slower and it tends to accumulate
into the current bloated mess that is the modern Web.  For static text
this really doesn't sound warranted.

(not to mention that mutating the body text after the HTML has loaded
may also produce a poor user experience, depending on various
conditions. And the native English speakers who develop the software
on top-grade machines will probably not notice it, thinking everything
is fine.)

As for your question, though, I would expect some of the less proficient
English speakers to also have outdated hardware or software installs,
especially in poor countries or very humble social environments.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Antoine Pitrou]

On Tue, 26 Jan 2016 06:50:15 -0500
Alexander Walters  wrote:
> Isnt the entire point of using centos 5 to use an ancient toolchain?

No, the point is to link against an ancient glibc. The toolchain can be
modern (and actually has to if you want to compile e.g. C++11 code).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Antoine Pitrou]

On Tue, 26 Jan 2016 20:36:26 +1000
Nick Coghlan  wrote:
> 
> If I understand the problem correctly, the CentOS 5 gcc toolchain is
> old enough that it simply doesn't emit the info libabigail needs in
> order to work.

If you build on CentOS 5, you certainly want to use the RH developer
toolset 2 which gives you a modern-ish toolchain (gcc 4.8.2, IIRC).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Antoine Pitrou]

On Thu, 21 Jan 2016 11:42:57 -0800
Chris Barker  wrote:
> nice, idea, but
> 
> libX11.so.6
> libXext.so.6
> libXrender.so.1
> libGL.so.1
> 
> These are all X11, yes? pretty much any workstation will have these, but in
> general, servers won't.

For the record, not having or not running a X11 server doesn't mean you
don't have a subset of the corresponding *libraries*.

For example, I have a Debian jessie server and, while there's no
X11-related executable on the machine, I still have the following
libraries installed:

$ dpkg -l | grep X
[...]
ii  libpangox-1.0-0:amd640.0.2-5   amd64
pango library X backend
ii  libpixman-1-0:amd64  0.32.6-3  amd64
pixel-manipulation library for X and cairo
ii  libpod-latex-perl0.61-1all  
module to convert Pod data to formatted LaTeX
ii  libx11-6:amd64   2:1.6.2-3 amd64
X11 client-side library
ii  libx11-data  2:1.6.2-3 all  
X11 client-side library
ii  libxau6:amd641:1.0.8-1 amd64
X11 authorisation library
ii  libxcb-render0:amd64 1.10-3+b1 amd64
X C Binding, render extension
ii  libxcb-shm0:amd641.10-3+b1 amd64
X C Binding, shm extension
ii  libxcb1:amd641.10-3+b1 amd64
X C Binding
ii  libxdmcp6:amd64  1:1.1.1-1+b1  amd64
X11 Display Manager Control Protocol library
ii  libxext6:amd64   2:1.3.3-1 amd64
X11 miscellaneous extension library
ii  libxft2:amd642.3.2-1   amd64
FreeType-based font drawing library for X
ii  libxml2:amd642.9.1+dfsg1-5+deb8u1  amd64
GNOME XML library
ii  libxpm4:amd641:3.5.11-1+b1 amd64
X11 pixmap library
ii  libxrender1:amd641:0.9.8-1+b1  amd64
X Rendering Extension client library
ii  xkb-data 2.12-1all  
X Keyboard Extension (XKB) configuration data
[...]

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New Design Landed in Warehouse

[Antoine Pitrou]

On Sat, 21 Nov 2015 08:46:45 +0100
Antoine Pitrou  wrote:
> 
> Agreed.  The look of the average project page is a bit depressing:
> https://warehouse.python.org/project/six/
> 
> Useful content starts only 2/3 down the first page. The large "pip
> install six" snippet probably doesn't deserve being that proeminent
> (or being there at all), and is ironically redundant with the "how do
> I install this?" link just below.
> 
> I would also suggest a bit more care on the typography :-) The main
> body text here:
> https://warehouse.python.org/project/requests/
> looks much less nice than here:
> https://pypi.python.org/pypi/requests/

To elaborate a bit and avoid misunderstandings, here are screenshots of
the aforementioned pages on my Web browser:
http://pitrou.net/warehouse.png
http://pitrou.net/pypi.png

Note how you get much more content on the current PyPI version, without
the page being too cluttered (some of the clutter actually comes from
the generic python.org navigation bar). Of course I'm not saying the
current PyPI layout is great and it shouldn't be replaced or overhauled,
but it does have that virtue of leaving most of the page area to the
project description.

(btw, I've shrank the warehouse font size a bit to make the comparison
fair, otherwise it would have been worse)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] New Design Landed in Warehouse

[Antoine Pitrou]

On Fri, 20 Nov 2015 11:40:23 -0500
Randy Syring 
wrote:

> I'm glad to see progress being made.  Thanks for the time and effort 
> that is being put into this.
> 
> After taking a look, the one thing that really stuck out to me in a 
> negative way was how much screen space the header is using up.  I've 
> created an issue for discussion here:

Agreed.  The look of the average project page is a bit depressing:
https://warehouse.python.org/project/six/

Useful content starts only 2/3 down the first page. The large "pip
install six" snippet probably doesn't deserve being that proeminent
(or being there at all), and is ironically redundant with the "how do
I install this?" link just below.

I would also suggest a bit more care on the typography :-) The main
body text here:
https://warehouse.python.org/project/requests/
looks much less nice than here:
https://pypi.python.org/pypi/requests/

(actually, in both those pages, it would be nice trying justifying the
text, as well)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Antoine Pitrou]

On Wed, 18 Nov 2015 06:47:53 +1300
Robert Collins  wrote:
> Note that previous PEPs have either had no grammar (and
> interop issue) or partially defined grammar's (and logical issues -
> see PEP-426 for example). I think its very important we be able to
> test what we're saying should happen well.

Of course. But only for the things the PEP is useful for: i.e.,
packaging-related information. Being able to separate valid URIs and
invalid URIs is not a useful feature for an implementation of this PEP.
Yet (correct me if I'm wrong) you seem to claim it's an integral part of
being a conformant implementation.

> Implementations don't have to use the grammar, they just have to
> accept the same inputs

It means they have to invoke some logic to reject invalid URIs upfront,
even though it's none of their business (since they will later treat
the URIs as strings, anyway).

> packaging's
> implementation doesn't use the same grammar, for instance. (It uses
> pyparsing and a mix of regexes and objects).

And the URI validation bit is implemented as...?

> Since the bit you're complaining about is basically an appendix, I can
> see that it makes the PEP shorter, but not how it makes it simpler: we
> still depend on the definition of URI, because we consume URI's

"We" don't depend on it. Whatever does the URI parsing and loading does
- another library certainly, not a packaging-specific library. From a
packaging point of view, URIs are plain strings, not something you
parse to extract meaningful information.  This is called abstraction :-)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Antoine Pitrou]

On Wed, 18 Nov 2015 06:32:35 +1300
Robert Collins  wrote:
> >
> > The only place where URIs are used seem to be the "urlspec" rule, and
> > probably you can accept any opaque string there.
> 
> Uhm, why are you making this suggestion? What problem will we solve by
> using a proxy rule?

Making the PEP simpler, making implementations simpler, avoiding bugs
in implementations which may otherwise try to implement full URI
matching, avoiding having to issue a PEP update whenever the IETF
updates the definition.

These are the four I can think about, anyway :-)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Antoine Pitrou]

On Wed, 18 Nov 2015 05:40:33 +1300
Robert Collins  wrote:
> 
> Its included in the complete grammar, otherwise it can't be tested.
> Note that that the PEP body refers to the IETF document for the
> definition of URIs. e.g. exactly what you suggest.

What I suggest is that the grammar doesn't try to define URIs at all,
and instead includes a simple rule that is a superset of URI matching.
It doesn't make sense for Python packaging tools to detect what is a
valid URI or not. It's not their job, and the work will probably be
replicated by whatever URI-loading library they use anyway (since they
will pass it the URI by string, not by individual components).

The only place where URIs are used seem to be the "urlspec" rule, and
probably you can accept any opaque string there.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Antoine Pitrou]

On Tue, 17 Nov 2015 09:33:56 -0500
Donald Stufft  wrote:
> 
> > On Nov 17, 2015, at 9:27 AM, Antoine Pitrou  wrote:
> > 
> >> 
> >> There are a number of separate subcommands that build systems must support.
> > 
> > I wonder how desirable and viable this all is. Desirable, because you
> > are still asking the build system to appear as setuptools *in some way*.
> > Viable, because pip may some day need to ask more from setuptools and
> > then third-party build tools will have to adapt and implement said
> > command-line options, defeating the "abstraction".
> > 
> > In other words, this PEP seems to be only solving a fraction of the
> > problem.
> 
> Can you explain this? I don’t see how it’s true. We need some way for pip
> to invoke the build system no matter what the build system is. Either that
> API is a Python API or that build system is a CLI based API but either way
> there needs to be some way for that to happen. This PEP chooses (at my
> request) a defined CLI API because it makes the delineation between build
> system and pip cleaner.

I may have misunderstood, it seemed to me that "wheel -d" and "develop"
are simply setuptools commands christened by the PEP.

I tend to think Python APIs are better than CLI APIs, but that probably
doesn't make a lot of difference.  This assumes of course that
potential problems are taken care of (such end-of-line conventions and
character encodings on stdin / stdout :-)).  The one of thing where a
CLI API is clearly inferior is error report, though...

> The whole point of this PEP is that once we have it, we can’t just randomly
> require more from the build tool than what is in the interface defined in
> this PEP. If we need more than we have to write a new PEP that extends the
> old interface with a new feature, but at all times it is built on an
> interface that is standardized via a PEP.

That clears things up, thank you.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] build system abstraction PEP, take #2

[Antoine Pitrou]

On Tue, 17 Nov 2015 15:24:04 +1300
Robert Collins  wrote:
> 
> The programmatic interface allows decoupling of pip from its current
> hard dependency on setuptools [#setuptools]_ able for two
> key reasons:
> 
> 1. It enables new build systems that may be much easier to use without
>requiring them to even appear to be setuptools.

And yet:

> There are a number of separate subcommands that build systems must support.

I wonder how desirable and viable this all is. Desirable, because you
are still asking the build system to appear as setuptools *in some way*.
Viable, because pip may some day need to ask more from setuptools and
then third-party build tools will have to adapt and implement said
command-line options, defeating the "abstraction".

In other words, this PEP seems to be only solving a fraction of the
problem.


> The file ``pypa.json`` acts as neutron configuration file for pip and other
> tools that want to build source trees to consult for configuration.

What is a "neutron configuration file"?

Why is it called "pypa.json" and not the more descriptive
"pybuild.json" (or, if you prefer, "pip.json")?
"pypa", as far as I know, is the name of an informal group, not a
well-known utility, operation or command.

> bootstrap_requires
> Optional list of dependency specifications [#dependencyspec] that must be
> installed before running the build tool.

How are the requirements gathered and installed? Using setuptools?

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Pip is not a library was: FINAL DRAFT: Dependency specifier PEP

[Antoine Pitrou]

On Tue, 17 Nov 2015 03:33:09 -0800
Nathaniel Smith  wrote:
> 
> Presumably there will be a dependency parser added to the 'packaging'
> library, which already exists as a standard place to stick stuff like this,
> so you'll just use that. (E.g. it's what pip uses for PEP 440 version
> parsing today.)
> 
> https://pypi.python.org/pypi/packaging

Ah... Is this a different thing than distlib? Does one depend on the
other?

(this may come to mind: https://www.jwz.org/doc/cadt.html :-))

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] FINAL DRAFT: Dependency specifier PEP

[Antoine Pitrou]

On Tue, 17 Nov 2015 09:46:21 +1300
Robert Collins  wrote:
> 
> URI_reference = 
> URI   = scheme ':' hier_part ('?' query )? ( '#' fragment)?
> hier_part = ('//' authority path_abempty) | path_absolute |
> path_rootless | path_empty
> absolute_URI  = scheme ':' hier_part ( '?' query )?
> relative_ref  = relative_part ( '?' query )? ( '#' fragment )?
> relative_part = '//' authority path_abempty | path_absolute |
> path_noscheme | path_empty
> scheme= letter ( letter | digit | '+' | '-' | '.')*
> authority = ( userinfo '@' )? host ( ':' port )?
> userinfo  = ( unreserved | pct_encoded | sub_delims | ':')*
> host  = IP_literal | IPv4address | reg_name
> port  = digit*
> IP_literal= '[' ( IPv6address | IPvFuture) ']'
> IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+
> IPv6address   = (
>   ( h16 ':'){6} ls32
>   | '::' ( h16 ':'){5} ls32
>   | ( h16 )?  '::' ( h16 ':'){4} ls32
>   | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32
>   | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32
>   | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32
>   | ( ( h16 ':'){0,4} h16 )? '::' ls32
>   | ( ( h16 ':'){0,5} h16 )? '::' h16
>   | ( ( h16 ':'){0,6} h16 )? '::' )

It seems weird that the PEP tries to include an entire subgrammar for
URIs, including even the parsing various kinds of IP addresses.  Why not
be lenient in their detection and leave actual definition of valid URIs
to the IETF?

It doesn't seem there is any point to embed/duplicate such knowledge in
Python packaging tools.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Mon, 9 Nov 2015 12:01:37 +
Paul Moore  wrote:
> 
> The one thing that *is* special about pip is that it actually
> *modifies* the Python installation it runs under.

Fortunately, though, if you are running the system pip without having
root privileges activated, it will most certainly fail with a
permission error.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Mon, 9 Nov 2015 11:16:56 +
Oscar Benjamin  wrote:
> On 9 November 2015 at 10:44, Wolfgang Maier
>  wrote:
> >
> > Something I miss in all the discussions taking place here is the fact that
> > python -m pip is the officially documented way of invoking pip at
> > https://docs.python.org/3/installing/index.html#basic-usage and it is not
> > particularly helpful if that recommendation keeps changing back and forth.
> >
> > I know some people don't like the wordy invocation, but other people
> > (including me) use and teach it because it works reliably. Just because a
> > pip executable based invocation pattern looks better, I don't think it
> > justifies the change.
> 
> I also teach this invocation. Somehow you have to select the Python
> version you're interested in and I really don't see why

"Selecting the Python version you're interested in", in many cases, is
done /a priori/ by activating the appropriate environment (whether
virtualenv-, pyvenv- or conda-based).

When I'm using a conda environment I certainly don't want to type
"python -m pip" when "pip" is sufficient.

It would be nice if people here could acknowledge the diversity of
existing workflows.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Sat, 7 Nov 2015 19:37:03 -0500
Donald Stufft  wrote:
> In fact, the pyvenv script has been deprecated and is going to be
> removed in Python 3.8 in favor of `python -m venv` for similar
> reasons that I've described here.

That's not an argument, since the decision was taken by exactly the
same people. Just because you do something twice doesn't mean it was a
good thing to do, especially when no sizable data was brought in
support.

The fact that you decided to deprecate "pyvenv" while its ancestor - the
"virtualenv" script itself - was not deprecated despite existing for a
much longer time hints that the issue may be blown out of proportion.

In the end, I like "python -m " for many user interfaces;
but for managing package installations I think it's really much too
wordy.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Sat, 7 Nov 2015 19:16:55 -0500
Donald Stufft  wrote:
> 
> The largest problem comes when ``python`` and ``pip`` disagree about which 
> Python is being invoked.

As a said, this is a problem for package managers and distributions.
"pip" isn't the only affected command, e.g. "pydoc" is as well.

> What should the command be to install into PyPy 2.4.0?

If you are using a virtualenv (or a conda environment, assuming you
did a conda package for pypy), just "pip".

> What if someone has /usr/bin/python2.7
> and /usr/bin/pip2.7 and they then install another Python 2.7
> into /usr/local/bin/python2.7 but they don’t have pip installed there?

Why wouldn't they? I thought the plan is to have "pip" bundled with
every recent Python version? AFAIR someone even said it was a bug if pip
wasn't installed together with Python...

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

On Sat, 7 Nov 2015 23:41:25 +
Paul Moore  wrote:
> On 7 November 2015 at 22:21, Antoine Pitrou  wrote:
> > The actual question is: which problem are you trying to solve *that
> > current users are actually experiencing*?
> 
> Typically, people using "pip" to install stuff, and finding it gets
> installed into the "wrong" Python installation (i.e., not the one they
> expected). I'm not clear myself on how this happens, but it seems to
> be common on some Linux distros (and I think on OSX as well) where
> system and user-installed Pythons get confused.

Well, the problem is that "python -m pip" isn't any better. If you
don't know what the current "pip" is, then chances are you don't know
what the current "python" is, either.

(I'm not trying to deny the issue, I sometimes wonder what "pip" will
install into exactly, but removing the command in favour of a "-m"
switch wouldn't do any any good IMO, and it would make Python package
management "even more baroque" than it currently is)

Regarrds

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Antoine Pitrou]

The actual question is: which problem are you trying to solve *that
current users are actually experiencing*?

I'm -1 on removing the "pip" command.  "python -m pip" is frankly not a
reasonable subtitution if we want to *promote* pip.

> * The above gets *really* confusing when ``pipX`` or ``pip`` do not agree with
>   what ``pythonX`` and ``python`` point to.

That's a problem for foreign package managers and distributions. Let
them deal with it.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ...

[Antoine Pitrou]

On Thu, 22 Oct 2015 18:15:37 +0200
Thomas Güttler  wrote:
> 
> > On top of this, the goal of lots of efforts around packaging is to allow 
> > people to move away from distutils/setuptools, as the underlying design is 
> > fundamentally difficult to extend.
> 
> If the underlying design is fundamentally difficult to extend, what should be 
> done?
> 
> Build tools on top of it? I guess this is not a good idea. 
> 
> Or start from scratch and deprecate the current underlying design in the 
> future?

Providing official hooks would be a way to provide extension facilities.
Of course this assumes someone would actually maintain distutils, and
"the community" would stop making a ruckus everytime an extremely minor
behaviour change is considered (otherwise you get what happened to
distutils2).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[Antoine Pitrou]

On Wed, 21 Oct 2015 21:41:35 +0100
Paul Moore  wrote:
> On 21 October 2015 at 20:41, Chris Barker  wrote:
> > As I understand it, the current "API" that pip knows about is:
> >
> > "some subset of what setuptools provides"
> >
> > So my proposal is here is to provide a way for users to easily use jsut that
> > subset.
> 
> https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface
> 
> All you need to do is write a setup.py, which can do *anything you
> like*,

Reading this, the CLI options which have to be implemented are
completely tied to setuptools' own view of the world.
`--single-version-externally-managed`? `--install-headers`? Why should
a random build system care about that gunk? What should it do with it?

I think Nathaniel's PEP, for all its shortcomings, looked much saner
than that piece of ad-hoc specification (a.k.a. "here's the random set
of things we're currently doing, let's make a spec out of it"). This is
like the Microsoft OOXML of Python packages distribution.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Time for a setuptools_lite??

[Antoine Pitrou]

On Wed, 21 Oct 2015 18:29:17 +
Daniel Holth  wrote:
> We're not just grumpy, we have seen The One Build System idea tried several
> times with bad results. Instead, if you have the inclination, time, and
> ability, you could try to build a new competing build system like
> http://flit.readthedocs.org/ did, or you could help try to figure out how
> to improve support for said competing build systems in pip, or you could
> even try to make a pip replacement that is better for distributing end-user
> applications.

That doesn't seem to follow at all. First sentence you're talking
about a build system, second you're talking about distributing end-user
applications.  Building and distributing are different things.  "flit"
certainly does nothing for non-trivial build chains.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Remove distutils, was: red, green, refactor ...

[Antoine Pitrou]

On Wed, 21 Oct 2015 17:05:29 +0200
Nick Coghlan  wrote:
> On 21 October 2015 at 14:55, David Cournapeau  wrote:
> >
> > On Wed, Oct 21, 2015 at 12:52 PM, Thomas Güttler
> >  wrote:
> >> ok, at the moment setuptools uses distutils.
> >>
> >> Why not melt them together into **one** underwear-pants-module?
> >
> >
> > What do you hope getting from that ? distutils is in the stdlib, so cannot
> > change easily, and even if putting setuptools in the stdlib were possible,
> > you would now need to handle different versions of setuptools for different
> > versions of python.
> 
> It's more useful to go the other direction and vendor a modern version
> of distutils inside setuptools:

It seems it would only add a bit more craziness to the current
landscape. What happens to projects which have a need to monkeypatch
distutils? How does it interact with the vendored version? etc.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Remove distutils, was: red, green, refactor ...

[Antoine Pitrou]

On Wed, 21 Oct 2015 15:30:51 +0300
Ionel Cristian Mărieș  wrote:
> On Wed, Oct 21, 2015 at 3:10 PM, Antoine Pitrou  wrote:
> 
> > (3) because distutils is part of the standard library and setuptools is
> > most likely unwelcome there :-)
> >
> 
> ​Well, it's not dead yet. Sorry, I mean,​ good enough to be included in the
> stdlib :-)
> 
> But seriously, is it correct to assume that it's not there for largely the
> same reasons pip ain't in stdlib? (and yes, getpip is in stdlib)

setuptools inclusion was discussed long ago AFAIK, and it was refused.
For pip the reasons are different (the much shorter release cycle, the
external dependencies required).

See e.g.:
https://mail.python.org/pipermail/python-dev/2006-April/063966.html
https://mail.python.org/pipermail/python-dev/2006-April/063886.html

(incidentally, the latter link shows a previous attempt to refactor
distutils was shot down because of backwards compatibility; which is
also why distutils2 was canned, despite bringing concrete positive
enhancements; had those attempts not been shot down, the situation
right now would be significantly better than it is...)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Remove distutils, was: red, green, refactor ...

[Antoine Pitrou]

On Wed, 21 Oct 2015 13:07:10 +0100
Paul Moore  wrote:
> On 21 October 2015 at 12:52, Thomas Güttler
>  wrote:
> > ok, at the moment setuptools uses distutils.
> >
> > Why not melt them together into **one** underwear-pants-module?
> 
> (1) because there's no-one with anything like the time or energy to do
> such a thing
> (2) because doing so would almost certainly break vast numbers of
> existing setup.py scripts

(3) because distutils is part of the standard library and setuptools is
most likely unwelcome there :-)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Wed, 14 Oct 2015 09:25:43 -0700
Chris Barker  wrote:
> On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou  wrote:
> 
> > On Tue, 13 Oct 2015 09:59:05 -0700
> > Chris Barker  wrote:
> > >
> > > So even is SSE2 provides little for Python itself, in the usual case,
> > we'll
> > > see performance hits n compiled extensions that are not compiled by
> > > particularly smart people.
> >
> > Since the question is only for 32-bit builds,
> 
> IS that the case:
> """
> Note that my recently retired computer was 64 bit and had SSE but didn't
> have SSE2 (I'm fairly sure - CPU was some budget AMD model)
> """
> 
> granted, such machines are probably really really rare, but maybe it does
> matter for 64 bit, too?

Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which
was originally devised by AMD), so I'm a bit skeptical about a
SSE2-less 64-bit CPU.  Do you have any reference?

> > does this even matter?
> > 32-bit builds on x86 generally bring you poorer performance by
> > themselves,
> 
> If a user has a 32 bit machine, they have no choice -- we could argue that
> anyone for whom performance matters probably isn't running an old, cheap
> machine, but still...

The actual question is whether we want to introduce a significant
amount of complexity and overhead for packagers and distributors, for
the benefit of an extremely small users demography that will probably
disappear altogether in a couple of years.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Tue, 13 Oct 2015 09:59:05 -0700
Chris Barker  wrote:
> 
> So even is SSE2 provides little for Python itself, in the usual case, we'll
> see performance hits n compiled extensions that are not compiled by
> particularly smart people.

Since the question is only for 32-bit builds, does this even matter?
32-bit builds on x86 generally bring you poorer performance by
themselves, as the CPU has only access to 8 general-purpose registers
(instead of 16) in that mode.  If you want better performance (or want
to efficiently address 3+GB of RAM, which is an extremely common
situation nowadays), you probably want 64-bit builds.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Sun, 11 Oct 2015 08:07:30 -0700
Steve Dower  wrote:
> 
> This does only affect 32-bit builds, so now I'm thinking about the
> possibility of treating those as highly compatible while the 64-bit
> ones get better performance treatment, though I'm not sure how that
> could actually play out. It may help remove some of the questions
> about which one to use though.

That sounds reasonable to me. I don't know Windows very much, but are
there still many people using 32-bit Windows these days (on x86, I
mean)?

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] sampleproject: use tox?

[Antoine Pitrou]

On Thu, 8 Oct 2015 22:05:16 +0200
Thomas Güttler  wrote:
> This is a follow up to the thread "Where should I put tests when packaging 
> python modules?"
> 
> I have never used tox up to now. But reading the mails of the thread, it seems
> that tox is the current state of the art.

I don't see why that would be the case.  tox is certainly "the state of
the art" in certain circles, but many projects don't mandate it at all
or even don't use it at all and rely on other mechanisms (e.g. CI with
a rich configuration matrix).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Wed, 7 Oct 2015 01:44:34 +0300
Ionel Cristian Mărieș  wrote:
> 
> That leaves me with 22 packages that need test tools like pytest/nose and
> assorted plugins.
> 
[...]
> 
> ​It's double the trouble to find compatible releases.

Hmm, are you saying py.test / nose and assorted plugins break APIs
often?  I would be a bit surprised, but I don't use them nowadays.
Modern unittest is quite capable.

> > And just because you are not used to a "workflow" doesn't make it
> > "weird" in any case.  Python itself uses such a workflow ("python -m
> > test").
> 
> ​It's weird in the sense that you have to do all these gymnastics to get
> the test dependencies right​

Well, "getting the dependencies right" must be done, whether you run
the tests from their installed location or from the source tree.  In
both cases, you can probably get the package management system to
automate package downloads and installs.

Rehards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 16:16:41 -0600
Carl Meyer  wrote:
> On 10/06/2015 04:04 PM, Antoine Pitrou wrote:
> [snip]
> > ...How are tests supposed to be a problem here, while they
> > usually have so few dependencies of their own?
> > 
> >> If you have to bend over backwards (to install the test dependencies)
> > 
> > While some packages may have non-trivial test dependencies, usual
> > practice is for test suites to require the exact same dependencies as
> > the rest of the package, + perhaps a test runner.
> > 
> > Since we're talking about good practice for the average package, it's
> > not very useful to point out that 0.1% of PyPI packages may have
> > excruciatingly annoying test dependencies.
> 
> I think this discussion could probably do with fewer unsupported
> assertions about what is "usual" -- it's clear that experiences in
> different parts of the community vary widely.
> 
> Speaking personally and anecdotally, I maintain 15 or so projects on
> PyPI, and every single one of them has at least three or four test-only
> dependencies; not just a test runner, but also testing utilities of one
> kind or another (e.g. the mock backport for Python 2).

They're still trivial dependencies, though.  Usually small or
medium-sized pure Python packages with a rather stable API
(especially stdlib backports, of course).  I don't see how they could
cause the kind of mess the OP claimed they would.

So I'd still like to know what "bend over backwards" is supposed to
mean here.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Wed, 7 Oct 2015 00:47:31 +0300
Ionel Cristian Mărieș  wrote:
> On Tue, Oct 6, 2015 at 11:54 PM, Erik Bray  wrote:
> 
> > Skimming back through the rest of the thread I don't see too much
> > support for 1).  The only argument against it is the need for
> > specifying dependencies, etc., which really only impacts developers so
> > long as the tests aren't *installed*, I think.  But there's also the
> > question of what kinds of tests we're talking about.  I think unit
> > tests should live in the .tests for a library.  Other
> > kinds of tests I don't have a strong opinion about.
> >
> 
> ​I think there's some confusion here. The pain point with "inside" tests is
> exactly the dependencies.

Is it your personal experience or some theoretical argument you're
making?

> If you install two packages with "inside" tests, then how do you deal with
> the version conflict of test dependencies?

Well, how do you deal with the version conflict of non-test
dependencies? How are tests supposed to be a problem here, while they
usually have so few dependencies of their own?

> If you have to bend over backwards (to install the test dependencies)

While some packages may have non-trivial test dependencies, usual
practice is for test suites to require the exact same dependencies as
the rest of the package, + perhaps a test runner.

Since we're talking about good practice for the average package, it's
not very useful to point out that 0.1% of PyPI packages may have
excruciatingly annoying test dependencies.

> Why completely mess up user's site-packages
> just because you want to have this weird `python -mfoobar.tests` workflow?

Did you have such an experience or are you making this up for the sake
of the argument?

And just because you are not used to a "workflow" doesn't make it
"weird" in any case.  Python itself uses such a workflow ("python -m
test").

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:33:03 -0400
Donald Stufft  wrote:
> On October 6, 2015 at 9:08:12 AM, Antoine Pitrou (solip...@pitrou.net) wrote:
> > On Tue, 6 Oct 2015 08:57:12 -0400
> > Donald Stufft wrote:
> > >
> > > It doesn't really make experimenting in a VCS any harder, since all you 
> > > need to
> > > do first is run ``pip install -e .`` and it will do a development install 
> > > and
> > > add the src/ directory to sys.path.
> >  
> > That means you're suddently polluting your Python install with a
> > development package. So either you create a dedicated virtualenv (more
> > command-line boilerplate, including each time you switch from a project
> > another) or you risk messing with other package installs.
> 
> Unless your project has zero dependencies you’ll want to use a dedicated 
> virtual environment anyways.

Not necessarily, you can share that environment with other projects.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 08:57:12 -0400
Donald Stufft  wrote:
> 
> It doesn't really make experimenting in a VCS any harder, since all you need 
> to
> do first is run ``pip install -e .`` and it will do a development install and
> add the src/ directory to sys.path.

That means you're suddently polluting your Python install with a
development package.  So either you create a dedicated virtualenv (more
command-line boilerplate, including each time you switch from a project
another) or you risk messing with other package installs.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 15:34:38 +0300
Ionel Cristian Mărieș  wrote:
> 
> Very few
> test runners change the current working directory by default [1], so it's
> better to just get a better project layout. pyca/cryptography
>  is a good example.​

The "src" convention is actually terrible when working with Python
code, since suddenly you can't experiment easily on a VCS checkout, you
have to do extra steps and/or write helper scripts for it.

The fact that few Python projects, including amongst the most popular
projects, use that convention mean it's really not considered a good
practice, nor convenient.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 07:07:31 -0400
Donald Stufft  wrote:
> 
> I've never, in my entire life [...]

Can I suggest your entire life is an anecdotal data point here?

> This is still ignoring the problems of test dependencies

Only if your tests have dependencies that runtime doesn't have.

> as well so you'll
> still need to ask them to install some number of dependencies, and I think 
> it's
> fairly trivial to ask someone to download a tarball, untar it, and run two
> commands.

Any number of things can be described as trivial depending on the
skillset and patience of the user.  When users report a bug, they are
not expecting to be asked to download and "untar" stuff.  Not every
user is a programmer.

> You're confusing "ships the test suite as part of the package" with "runs the
> test suite on the installed package". The two aren't really related, you can
> run the tests against an installed package trivially in either situation.

It's not trivial, because if you aren't careful you'll be running them
against the tarball / checkout instead (because of Python munging the
PYTHONPATH behind your back, for example), and this can go unnoticed for
a long time. By contrast, if you don't need a tarball / checkout to run
them, you can guarantee they are run against the installed location.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 11:30:00 +0300
Ionel Cristian Mărieș  wrote:
> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou  wrote:
> 
> > They should be inside the module. That way, you can check an installed
> > module is ok by running e.g. "python -m mypackage.tests". Any other
> > choice makes testing installed modules more cumbersome.
> >
> 
> ​Does that really make sense? I haven't heard of any user actually running
> tests​
> that way. To be honest I haven't ever ran Python's own tests suite as part
> of a user installation.

There are several situations besides the "downstream packagers" use case
mentioned somewhere else:

* One of your users report a weird issue, you can ask them to run the
  test suite on their installation to check that nominal behaviour of
  the package is ok on their machine.  If you don't ship the test suite,
  you have to ask them to do extra manual steps in order to do this
  verification, which can be cumbersome and delay proper response to
  the issue.

* Your package requires non-Python data files for proper functioning,
  and you want to check the installation procedure puts them in the
  right place.  The natural way to do that is to run the test suite on
  the installed package.

Really, "ship the test suite" should be the norm and not shipping it
should be the exception (if e.g. testing needs large data files).

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:17:22 +0100
Paul Moore  wrote:
> On 6 October 2015 at 08:51, Antoine Pitrou  wrote:
> > They should be inside the module. That way, you can check an installed
> > module is ok by running e.g. "python -m mypackage.tests". Any other
> > choice makes testing installed modules more cumbersome.
> 
> One inconvenience with this is that if you use an external testing
> framework like nose or pytest, you either need to make your project
> depend on it, or you need to document that "python -m mypackage.tests"
> has additional dependencies that are not installed by default.
> 
> With an external tests directory, the testing framework is just
> another "development requirement".

Doesn't / didn't setuptools have something called test_requires?

> It would also be very easy to take the view that the PyPA sample
> project should omit the test directory altogether, as it's a sample
> for the *packaging* guide, and development processes like testing are
> out of scope (that's why we don't include a documentation directory,
> or recommend sphinx, for example).

That sounds like the best course to me.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Where should I put tests when packaging python modules?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:07:46 +0200
Thomas Güttler  wrote:
> 
> Dear experts, please decide:
> 
> inside the module like this answer:
> 
>  
> http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules

They should be inside the module. That way, you can check an installed
module is ok by running e.g. "python -m mypackage.tests". Any other
choice makes testing installed modules more cumbersome.

> outside the module like this:
> 
>  https://github.com/pypa/sampleproject/tree/master/tests

There is no actual reason to do that except win a couple kilobytes if
you are distributing your package on floppy disks for consumption on
Z80-based machines with 64KB RAM.

Even Python *itself* puts its test suite inside the standard library,
not outside it (though some Linux distros may strip it away).
Try "python -m test.regrtest" (again, this may fail if your distro
decided to ship the test suite in a separate package).

The PyP"A" should definitely fix its sample project to reflect good
practices.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Antoine Pitrou]

On Mon, 5 Oct 2015 09:51:05 -0400
Donald Stufft  wrote:
> 
> You can see these needs are different I think by looking at how what 
> Nathaniel wants differs from what me and Paul want. He wants something that 
> will make the human side easier and will support different tools, we want 
> something that pip can consume more reasonably. Trying to do too much with a 
> singular format just means it sucks for all the uses cases instead of being 
> great for one use case. 

That doesn't seem to follow. You can have regular human-compatible
content and a few machine-compatible files besides (perhaps in a
dedicated subdirectory). I don't see how that "sucks".

> I also don't think it will be confusing. They'll associate the VCS thing (a 
> source release) as something focused on development for most everyone. Most 
> people won't explicitly make one and nobody will be uploading it to PyPI.

Well, what is the point of standardizing the concept of source releases
if nobody produces them?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Antoine Pitrou]

On Mon, 5 Oct 2015 09:28:48 -0400
Donald Stufft  wrote:
> > 
> > In any case, sdists being apt for human consumption is an important
> > feature IMHO.
> > 
> 
> I don't think so. Remember in my footnote I mentioned that I wasn't using VCS
> to mean literally "something checked into a VCS", but rather the more
> traditional (outside of Python) "source release" concept. This could be a
> simple tarball that just has the proper, for human consumption, files in it
> or it could be a VCS checkout, or it could just be some files sitting on disk.

But why use two different formats for "source release" and "sdists"?
Currently sdists fit the assumptions for a source release, why
introduce some complexity and have the users deal with separate
concepts (with all the confusion that will inevitably ensue)?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Antoine Pitrou]

On Mon, 5 Oct 2015 08:44:04 -0400
Donald Stufft  wrote:
> 
> I don't think that it makes sense for pip to go directly from a VCS[1] to a
> Wheel in the grand scheme of things. Right now we kind of do it, but that's
> because we just treat them like an unpacked sdist [2], long term though I 
> don't
> think that is the correct way to do things. We (I?) want to minimize the
> different installation "paths" that can be taken and the main variable then
> when you do a ``pip install`` is how far long that path we already are. My
> ideal path looks something like this:
> 
>     VCS -> Source Wheel [3] -> Wheel -> Installed
>     \-> Inplace Installation [4]

A valid use case may be to do an in-place installation from a sdist
(although you may question the sanity of doing development from a
source tree which isn't VCS-backed :-)).

In any case, sdists being apt for human consumption is an important
feature IMHO.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Antoine Pitrou]

On Mon, 5 Oct 2015 14:11:56 +0100
Paul Moore  wrote:
> 
> Being a Windows user, I hadn't caught the parallel between
> Wheel-Source wheel and RPM-Source RPM (is there also a similar
> deb-source deb pair?)
> 
> But if the concept works for people with a Linux background, I'm OK with it.

The "source RPM" concept is actually confusing since, IIUC, a
"source RPM" is nothing like a normal RPM (i.e. you can't install it
using the "rpm" tool). It should actually be called a "RPM
source" (again, IIUC).

> (My main concern is that end users commonly make requests to projects
> saying "please provide wheels". I don't want that nice simple concept
> to get confused, if we can avoid it).

+1

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Bogus search result

[Antoine Pitrou]

Hello,

I don't if that's just me, but I go to https://pypi.python.org/pypi and
type "llvmlite" into the search box, clicking the "search" button leads
me to the page for the Numba package.  It should either lead me to the
page for the *llvmlite* package or show me a list of search results.

Regards

Antoine.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Reviving PEP 470 - Removing External Hosting support on PyPI

[Antoine Pitrou]

On Wed, 26 Aug 2015 21:24:05 -0400
Donald Stufft  wrote:
> 
> At the time of this writing there are 65,232 projects hosted on PyPI and of
> those, 59 of them rely on external files that are safely hosted outside of 
> PyPI
> and 931 of them rely on external files which are unsafely hosted outside of
> PyPI. This shows us that 1.5% of projects will be affected in some way by this
> change while 98.5% will continue to function as they always have. In addition,
> only 5% of the projects affected are using the features provided by PEP 438 to
> safely host outside of PyPI while 95% of them are exposing their users to
> Remote Code Execution via a Man In The Middle attack.

Out of curiosity, have you tried to determine if those Unsafely Off
PyPI projects were either still active or "popular" ?

The PEP looks fine anyway, good job :)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP for dependencies on libraries like BLAS (was: Re: Working toward Linux wheel support)

[Antoine Pitrou]

On Sat, 22 Aug 2015 20:33:24 +1000
Nick Coghlan  wrote:
> 
> Patience is one of the hardest development skills to learn (I
> regularly struggle with it myself), but when we don't apply ourselves
> to that task, not only do we significantly increase the likelihood of
> burning ourselves out, we're also likely to seriously annoy the folks
> around us.

Note the etymology of "patience" ;-)

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Antoine Pitrou]

On Thu, 20 Aug 2015 15:40:57 -0400
Nate Coraor  wrote:
> >
> > In practice, the `platform` module does not really keep up to date with
> > evolution in the universe of Linux distributions.
> >
> 
> Understandable, although so far it's doing a pretty good job:

Hmm, perhaps that one just parses /os/lsb-release, then :-)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Antoine Pitrou]

On Thu, 20 Aug 2015 14:26:44 -0400
Nate Coraor  wrote:
> 
> So I need a bit of guidance here. I've arbitrarily chosen some tags -
> `rhel` for example - and wonder if, like PEP 425's mapping of Python
> implementations to tags, a defined mapping of Linux distributions to
> shorthand tags is necessary (of course this would be difficult to keep up
> to date, but binary-compatibility.cfg would make it less relevant in the
> long run).
> 
> Alternatively, I could simply trust and normalize
> platform.linux_distribution()[0],

In practice, the `platform` module does not really keep up to date with
evolution in the universe of Linux distributions.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Antoine Pitrou]

On Fri, 17 Jul 2015 08:36:39 -0700
Chris Barker  wrote:
> 
>  - Packages with non-standard non-python dependencies: libhdf5, lapack,
> BLAS, fortran(!) -- this is where the nightmare really is. I suspect most
> folks on this list will say that this is "Scipy Problem", and indeed,
> that's where the biggest issues are, and where systems like conda have
> grown up to address this.
> 
> But at this point, I think it's really sad that the community has become
> fractured -- if folks start out with "I want to do scientific computing",
> then they get pointed to Enthought Canopy or Anaconda, and all is well
> (until they look for standard web development packages -- though that's
> getting better). But if someone starts out as a web developer, and is all
> happy with the PyPA stack (virtualenv, pip, etc...), then someone suggests
> they put some Bokeh plotting in their web site, or need to do
> some analytics on HDF5 files, or any number of things well supported by
> Python, but NOT by pip/wheel -- they are kind of stuck.

Indeed, that's the main issue here. Eventually some people will want to
use llvmlite or Numba in an environment where there's also a web
application serving stuff, or who knows other combinations.

> PS: Personally, after banging my head against this for years,
> I've committed to conda for the moment -- working to get conda to better
> support the wide range of python packages. I haven't tried it on Linux, but
> it does exist and works well for some folks.

Due to the fact Linux binary wheels don't exist, conda is even more
useful on Linux...

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Thu, 9 Jul 2015 17:52:06 +0100
David Cournapeau  wrote:
> I don't think it is reasonable for pypa to recommend one solution when
> multiple are available (though it is certainly fair to mention them).
> 
> ActiveState, Enthought (my own employer) also provide linux binaries,

You are right, I was forgetting about them. Then mentioning them would
probably work.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Thu, 9 Jul 2015 23:50:30 +1000
Nick Coghlan  wrote:
> 
> As Donald notes, I think we're now in a good position to start making
> progress here, but the first step is going to be finding a way to
> ensure that *by default*, pip on Linux ignores wheel files published
> on PyPI, and requires that they be *whitelisted* in some fashion
> (whether individually or categorically). That way, we know we're not
> going to make the default user experience on Linux *worse* than the
> status quo while we're still experimenting with how we want the
> publication side of things to work. Debugging build time API
> compatibility errors can be hard enough, debugging runtime A*B*I
> compatibility errors is a nightmare even for seasoned support
> engineers.

By the way, I think there's another possibility if the Python packaging
authority doesn't want to tackle this (admittedly delicate) problem:
issue a public statement that Anaconda is the preferred way of
installing Linux binary packages if they aren't provided (or the
version is too old) by their Linux distribution of choice.

It would then give more authority to software developers if they want
to tell their users "don't use pip to install our code under Linux, use
conda".

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Wed, 08 Jul 2015 13:05:45 -0400
Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 07/08/2015 07:10 AM, Antoine Pitrou wrote:
> 
> > Seriously, how this is even supposed to be relevant? The whole point
> > is to produce best-effort packages that work on still-supported
> > mainstream distros, not any arbitrary "Linux" setup.
> 
> I'm arguing that allowing PyPI uploads of binary wheels for Linux will be
> actively harmful.

There is no point in reinstating an argument that has already been made
and discussed in the other subthread (of course, you would have to read
it first to know that).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Wed, 08 Jul 2015 05:39:13 -0400
Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 07/07/2015 07:43 PM, Antoine Pitrou wrote:
> 
> > That's a dramatically uninformed statement, to put it politely...
> > Some packages have difficult-to-meet build dependencies, and can also
> > take a long time to do so.
> 
> In the general case, it is *exactly* those projects which are going to
> trip people up when you upload their binary wheels to PyPI:  there will
> be no way to know that the compiled-in stuff will work on "any" Linux,
> and solving the problem of "which" Linux variants a given wheel can
> support is intractable.

Seriously, how this is even supposed to be relevant? The whole point is
to produce best-effort packages that work on still-supported mainstream
distros, not any arbitrary "Linux" setup.

Instead of lecturing people about what is in your opinion
"intractable", how about you just shut up, if you don't have anything
constructive to contribute?

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Tue, 07 Jul 2015 19:32:37 -0400
Tres Seaver  wrote:
> 
> Compared to Windows, and even somewhat OS/X, the win for uploading wheels
> to PyPI is miniscule:  pretty much everybody developing on Linux has or
> can get the toolchain required to build a wheel from an sdist, and can
> share those built wheels across the hosts she knows to be compatible.

That's a dramatically uninformed statement, to put it politely...  Some
packages have difficult-to-meet build dependencies, and can also take a
long time to do so. 

llvmlite, the package I'm talking about, builds against the
development libraries for LLVM 3.6 (a non-trivial download and install,
assuming you can find binaries of that LLVM version for your OS
version otherwise, count ~20 minutes to compile it with a modern
quad-core CPU).  We regularly have bug reports from people failing to
compile the package on Linux (and OS X), which is why we are considering
the option of pre-built binary wheels (in addition to the conda
packages we already provide, and which some people are reluctant to
use).

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Tue, 7 Jul 2015 11:02:40 -0400
Donald Stufft  wrote:
> In my mind, the biggest reason to not just open up the ability to upload even
> generic linux wheels right now is the lack of a safe-ish default. I think if
> we added a few things:
> 
> * Default to per platform tags (e.g. ubuntu_14_04), but allow this to be
>   customized and also accept "Generic" Linux wheels as well.

That would be cool :)

> * Put the libc into the file name as well since it's reasonable to build a
>   "generic" linux wheel that statically links all dependencies (afaik), 
> however
>   it does not really work to statically link glibc.

True. For example, here is the meat of a build of llvmlite on Linux:

$ ldd 
miniconda3/pkgs/llvmlite-0.6.0-py34_5/lib/python3.4/site-packages/llvmlite/binding/libllvmlite.so
linux-vdso.so.1 =>  (0x7ffeacefd000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f8c9e2f5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x7f8c9e0d7000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f8c9decf000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f8c9dccb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f8c9d9c5000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x7f8c9d7ae000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f8c9d3ea000)
/lib64/ld-linux-x86-64.so.2 (0x7f8c9fcee000)

It embeds LLVM and has no dynamic reference to anything beside the most
basic runtime libraries (libstdc++ is statically linked in).  The .so
file doesn't even require Python (but the rest of llvmlite, which loads
the .so file using ctypes, does need Python, of course).

We use a similar strategy on Windows and OS X.

> This means that even if
>   you build against an old version of glibc if you're running on a Linux that
>   *doesnt* use glibc (like Alpine which uses MUSL) you'll run into problems.

glibc vs. non-glibc is yet a different concern IMHO. Mainstream Linux
setups use glibc.

> You have to ensure all your
> dependencies are statically linked (if you have any) and you have to ensure
> that you build against an old enough Linux (likely some form of CentOS).

Yes, we use CentOS 5...

> * Side question, since I don't actually know how a computer works: Is it even
>   possible to have a CPython extension link against a different libc than
>   CPython itself is linked against?

Half-incompetent answer here: I think link-time it would be fine. Then
at library load time it depends on whether the actual system glibc is
ABI/API-compatible with the one the C extension (and/or CPython itself)
was linked with.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Tue, 7 Jul 2015 23:53:59 +1000
Nick Coghlan  wrote:
> On 7 July 2015 at 07:46, Antoine Pitrou  wrote:
> > On Mon, 6 Jul 2015 22:34:38 +0100
> > Paul Moore  wrote:
> >> On 6 July 2015 at 19:18, Antoine Pitrou  wrote:
> >> > What if packagers take care of working around the issue?
> >> > (for example by building on a suitably old Linux platform, as we
> >> > already do for Conda packages)
> >>
> >> At the moment it's just a simple "if the wheel is for Linux, reject it" 
> >> test.
> >>
> >> As to whether that's too conservative, one of the Linux guys would
> >> need to comment. Maybe the issue is simply that we can't be sure
> >> people will take the care that you do, and the risk of people getting
> >> broken installs is too high?
> >
> > Then how about a warning, or a rejection by default with a well-known
> > way to bypass it?
> 
> Unfortunately, the compatibility tagging for Linux wheels is currently
> so thoroughly inadequate that even in tightly controlled environments
> having a wheel file escape from its "intended" target platforms can
> cause hard to debug problems.

I'm not sure what you're pointing to, could you elaborate a bit?

For the record, building against a well-known, old glibc + gcc has
served the Anaconda platform well.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Mon, 6 Jul 2015 22:34:38 +0100
Paul Moore  wrote:
> On 6 July 2015 at 19:18, Antoine Pitrou  wrote:
> > What if packagers take care of working around the issue?
> > (for example by building on a suitably old Linux platform, as we
> > already do for Conda packages)
> 
> At the moment it's just a simple "if the wheel is for Linux, reject it" test.
> 
> As to whether that's too conservative, one of the Linux guys would
> need to comment. Maybe the issue is simply that we can't be sure
> people will take the care that you do, and the risk of people getting
> broken installs is too high?

Then how about a warning, or a rejection by default with a well-known
way to bypass it?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

On Mon, 6 Jul 2015 19:03:19 +0100
Paul Moore  wrote:
> On 6 July 2015 at 17:24, Antoine Pitrou  wrote:
> > (yes, the version number is off - but that's besides the point here,
> > I think):
> >
> > $ twine upload dist/llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl
> > /home/antoine/.local/lib/python3.4/site-packages/pkginfo/installed.py:53: 
> > UserWarning: No PKG-INFO found for package: pkginfo
> >   warnings.warn('No PKG-INFO found for package: %s' % self.package_name)
> > Uploading distributions to https://pypi.python.org/pypi
> > Uploading llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl
> > HTTPError: 400 Client Error: Binary wheel for an unsupported platform
> 
> PyPI does not support uploading binary wheels for Linux. This is a
> deliberate restriction because the tags supported by the wheel spec
> are not fine enough grained to specify compatibility between the
> myriad of Linux variations. The intention is that once a resolution to
> that problem is found, this restriction will be lifted.

What if packagers take care of working around the issue?
(for example by building on a suitably old Linux platform, as we
already do for Conda packages)

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] 400 Client Error: Binary wheel for an unsupported platform

[Antoine Pitrou]

Hello,

I'm having the following errors when trying to upload a wheel to PyPI.
(yes, the version number is off - but that's besides the point here,
I think):

$ twine upload dist/llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl 
/home/antoine/.local/lib/python3.4/site-packages/pkginfo/installed.py:53: 
UserWarning: No PKG-INFO found for package: pkginfo
  warnings.warn('No PKG-INFO found for package: %s' % self.package_name)
Uploading distributions to https://pypi.python.org/pypi
Uploading llvmlite-0.0.0-py2.py3-none-linux_x86_64.whl
HTTPError: 400 Client Error: Binary wheel for an unsupported platform

$ twine upload dist/llvmlite-0.0.0-py3-none-linux_x86_64.whl 
/home/antoine/.local/lib/python3.4/site-packages/pkginfo/installed.py:53: 
UserWarning: No PKG-INFO found for package: pkginfo
  warnings.warn('No PKG-INFO found for package: %s' % self.package_name)
Uploading distributions to https://pypi.python.org/pypi
Uploading llvmlite-0.0.0-py3-none-linux_x86_64.whl
HTTPError: 400 Client Error: Binary wheel for an unsupported platform


Thanks

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PyPI search engine slow

[Antoine Pitrou]

Hello,

It seems that PyPI's search engine has become quite slow. For example
the following URL takes ~8s. to load:

https://pypi.python.org/pypi?%3Aaction=search&term=signature%20inspect&submit=search

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Role of setuptools and eggs in "modern" distributing...

[Antoine Pitrou]

On Tue, 23 Dec 2014 09:36:36 -0800
Chris Barker  wrote:
> 
> So far, we've been doing mostly pip and struggling with build our own for
> the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL,
> and). But at the end of all this we'd like to be able to distribute and
> make it easy on end users to use our tools.
> 
> I figure we we'll do one (or both) of:
> - providing a custom "wheel house" with our packages and the dependencies
> that are hard to come by
> - provide a binstar channel with conda packages of all the same stuff but a
> totally different set of "other" packages.
> 
> At the moment, I'm working on making conda packages, which brings me to my
> questions.

Note you can use pip inside conda environments, which works quite well
at least for pure Python packages (start with "conda install pip").

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PYD/SO platform tags (#22980)

[Antoine Pitrou]

On Fri, 12 Dec 2014 23:23:03 +
Steve Dower  wrote:

> Hi distutils-sig,
> 
> There's a bit of discussion going on at http://bugs.python.org/issue22980 
> about extending SO tags to include bitness values, and I took the opportunity 
> to look into adding platform tags for .pyd files on Windows.
> 
> There's a patch up there, but I'm interested in any thoughts or concerns 
> people may have that I haven't considered. Distribution is likely to be most 
> affected, and I figured distutils-sig is most likely to have people with 
> strong opinions on the subject anyway, so I'm posting here :)
> 
> Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll 
> go from [".pyd"] to [".cp35-win32.pyd", ".cp3-win32.pyd", ".pyd"] (where 
> "win32" may also be "win_amd64", "win_ia64" or "win_arm", which are the 
> platforms that have #ifdefs in pyconfig.h).

For the record, https://www.python.org/dev/peps/pep-3149/#pep-384
suggests "abi3" as the ABI tag for the Python 3 stable ABI.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Antoine Pitrou]

On Sun, 30 Nov 2014 03:02:57 +1000
Nick Coghlan  wrote:
> Many users (quite reasonably, if they're primarily Python developers)
> have problems working through build failures when attempting to
> install non-Python extensions from source. Such build failures are
> usually models of clarity compared to diagnosing dynamic linking
> failures.

However, installing a binary doesn't imply a potential longish building
step, or the installation of many build dependencies. LLVM can take 20
minutes to compile on a modern quad-core x86. I've been told it takes
several hours on a Cortex A8 platform... By comparison, the failure of
loading a precompiled dynamic library is instantaneous.

And I don't think build failures are understandable by many users. You
need to be a seasoned C developer for that.

> > (at Continuum we have started offering such a service, but it's
> > "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix)
> 
> Yes, Continuum avoided the distro ABI compatibility problem by
> defining its own ABI.

Not exactly. Some ABI problems - for example the glibc-related ones -
are still here. Conda and binstar-build are still a best effort (on the
GNU/Linux side, that is), not an ideal solution.

> > Well, *allowing* distro tags in the platform tag is certainly ok. What
> > I'm afraid of is if that's made mandatory.
> 
> OK, that makes more sense. Yes, I agree we need to keep the ability to
> say "this is a prebuilt, self-contained, binary wheel that should run
> on any Linux system because it doesn't link to any system binaries".
> Chalk it up as yet another reason that the specific proposal I started
> the thread with wouldn't actually work :)

Great!

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Antoine Pitrou]

On Sun, 30 Nov 2014 01:47:16 +1000
Nick Coghlan  wrote:
> On 29 November 2014 at 01:51, Antoine Pitrou  wrote:
> > On Sat, 29 Nov 2014 01:27:44 +1000
> > Nick Coghlan  wrote:
> >> >
> >> > Is this not going to be a slippery slope?
> >>
> >> Only if folks publish Linux binaries themselves, and that's still a
> >> bad idea (for the same reason publishing distro binaries is already a
> >> rare thing for people to do).
> >
> > Well, let's not make this a matter of ideology. Everyone knows it's a
> > bad idea to publish binaries, yet it's often better than nothing,
> > especially if the software is tedious to compile.
> 
> It's not a matter of ideology, but a matter of practicality. Debian
> stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow
> enough (and have strong enough ABI compatibility guarantees) to be
> practical for ISVs to target with prebuilt binaries.

It seems we disagree on the notion of "practicality" :-)

For me practicality means being able to build a single binary package
for all recent Linux distros in a best effort approach. Building a
different package for each distro version is far from practical for any
reasonable-sized project (i.e. not something sponsored by a
1000+-employee entity, with a dedicated build team and infrastructure).

> > Case in point: can I ask you (the mythical "we") to build packages for
> > all major distros (including supported LTS releases), and the four most
> > recent Python versions, of the following piece of software:
> > https://github.com/numba/llvmlite
> 
> No, that would be a service provided by the as yet hypothetical PyPI
> build farm. If/when that happens, it will need to have a way of
> tagging Linux wheels appropriately, though.

"If/when that happens" is not reassuring, especially in the light of
how many pie-in-the-sky improvements in the packaging ecosystems have
turned out :-/

(at Continuum we have started offering such a service, but it's
"generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix)

> Nearer term (and what prompted me to start this thread), the Fedora
> Environments & Stacks working group is investigating providing
> prebuilt wheel files for the Fedora ecosystem, and potentially for
> EPEL as well (see
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
> for the broader context of that effort). For other ecosystems, you'll
> have to ask participants in those ecosystems.

That's asking software authors to complicate and slow down their
development process a lot. Also, there's no guarantee that Fedora or
Ubuntu or whatever would actually *accept* to help us, right?

> > I'm not sure I understand: distros provide their own packages, they
> > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they
> > depend on the wheel tagging format?
> 
> We don't plan to blindly pull anything from PyPI - we're looking at
> the feasibility of publishing already reviewed software in ecosystem
> native formats (with the two pilot projects focusing on Java JAR files
> and Python wheel files).
> 
> When I last mentioned that idea here, Marcus pointed out that doing
> that with the generic "linux_x86_64" compatibility tag on the wheel
> filenames would be problematic, as there'd be nothing preventing
> anyone from pulling them down onto inappropriate systems, with no
> obvious trace of the Fedora or EPEL specific assumptions that went
> into building them.

Uh, there's a lot of hidden knowledge required to understand those
two paragraphs that I don't master. I don't know what "inappropriate
systems" are, what are "reviewed software", etc. ;-)

Also I don't understand why you're not recompiling as you would
normally do.

> While that's a valid concern, I also don't want to go invent our own
> custom compatibility tagging convention just for Fedora & EPEL, but
> rather work within the limits of what upstream Python packaging
> natively supports.

Well, *allowing* distro tags in the platform tag is certainly ok. What
I'm afraid of is if that's made mandatory.

> Consider if the following could be included in the "pyvenv.cfg" file
> in a virtual environment:
> 
> [compatibility]
> python=cp27,cp2,py2
> abi=cp27mu
> platform=linux_x86_64_epel_6
> 
> Or for a Python 3 virtual environment:
> 
> [compatibility]
> python=cp34,cp3,py3
> abi=cp34m,abi3
> platform=linux_x86_64_epel_6
> 
> If present, these pyvenv.cfg settings would override the normal PEP
> 425 compatibility tag calculations (I'm OK with the idea of *needing*
> to be in a virtual environment to gain the power to configure these
> tags).

As long as it's on an opt-in basis, it certainly sounds ok.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Antoine Pitrou]

On Sat, 29 Nov 2014 01:27:44 +1000
Nick Coghlan  wrote:
> >
> > Is this not going to be a slippery slope?
> 
> Only if folks publish Linux binaries themselves, and that's still a
> bad idea (for the same reason publishing distro binaries is already a
> rare thing for people to do).

Well, let's not make this a matter of ideology. Everyone knows it's a
bad idea to publish binaries, yet it's often better than nothing,
especially if the software is tedious to compile.

> > How many binary packages will package authors have to provide to cover
> > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied
> > by three or four Python versions is already a lot of binaries to
> > build...
> 
> I'd still advise against folks posting Linux wheels on PyPI, just as
> they tend not to post RPM or deb files. This is so we can provide
> wheels at the distro level (or build them internally) without creating
> vast amounts of confusion.

So do we (software authors) have to wait for that mythical "we" who are
going to build binaries in time for our packages?

Case in point: can I ask you (the mythical "we") to build packages for
all major distros (including supported LTS releases), and the four most
recent Python versions, of the following piece of software:
https://github.com/numba/llvmlite

:-)

> This isn't really about that - it's about having a way to
> tackle it at the distro level, without introducing significant
> potential for confusion on end user systems

I'm not sure I understand: distros provide their own packages, they
don't (shouldn't) blindly pull binary wheels from PyPI. Why would they
depend on the wheel tagging format?

> The difference isn't really that surprising - both Microsoft and Apple
> have relied heavily on intellectual monopoly laws to retain control of
> their ecosystems. You can do a lot to constrain the choices of others
> when you have the full weight of the US government and copyright
> industry behind you.

That discussion is a bit off-topic, but I don't think it has anything
to do with copyright (and from I've seen in python-dev discussions,
I'm not sure Apple is a good example).

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Antoine Pitrou]

On Fri, 28 Nov 2014 16:03:59 +1000
Nick Coghlan  wrote:
> Here's my proposed change:
> 
> =
> The default platform tag is distutils.util.get_platform() with all
> hyphens - and periods . replaced with underscore _ . If
> /etc/os-release [N] exists on the system, then the values in the 'ID'
> and 'VERSION_ID' fields are read, all hyphens - and periods . replaced
> with underscore _ , and the results appended to the default tag after
> a separating underscore."
> 
> Examples:
> 
> * win32
> * macosx_10_6_intel
> * linux_x86_64_fedora_20
> * linux_x86_64_rhel_7_0
> * linux_x86_64_debian_7_0
> * linux_x86_64_ubuntu_14_04

Is this not going to be a slippery slope?

> Now, this slightly overspecifies on the *consumer* side. A binary
> wheel that works on "rhel_7_0" for example, should almost certainly
> work on "rhel_7_1". However, that can be addressed on the tooling side
> (e.g. permitting the specification of "additional compatible
> platforms" when invoking pip), rather than needing to be in the
> specification.

How about those lesser known distributions (e.g. Linux Mint or Mageia)?
How many binary packages will package authors have to provide to cover
people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied
by three or four Python versions is already a lot of binaries to
build...

While this would be a good technical solution, I think it's socially
disastrous.

Of course, you may point out that it has its roots in the failure of the
GNU/Linux ecosystem to provide real binary compatibility. It's stunning
that under Windows you can build a Windows XP-compatible shared library
with a recent MSVC just by turning a switch in the options...

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[Antoine Pitrou]

On Fri, 7 Nov 2014 15:46:36 +
Paul Moore  wrote:
> I'm in the process of developing an automated solution to allow users
> to quickly set up a Windows box so that it can be used to compile
> Python extensions and build wheels. While it can obviously be used by
> Windows developers who want to quickly set up a box, my main target is
> Unix developers who want to provide wheels for Windows users.
> 
> To that end, I'd like to get an idea of what sort of access to Windows
> a typical Unix developer would have. I'm particularly interested in
> whether Windows XP/Vista is still in use, and whether you're likely to
> already have Python and/or any development tools installed. Ideally, a
> clean Windows 7 or later virtual machine is the best environment, but
> I don't know if it's reasonable to assume that.

I use a Windows 7 VM with SP1. All running under KVM with virt-manager.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] "not a supported wheel on this platform"

[Antoine Pitrou]

On Tue, 28 Oct 2014 15:03:23 +
Paul Moore  wrote:
> On 28 October 2014 14:51, Antoine Pitrou  wrote:
> > Does matching really happen that way?
> 
> Unfortunately yes :-(
> 
> get_supported() introspects the system you're installing on and builds
> a list of tag combinations it's willing to accept. In doing so it
> misses some that I'd consider obscure but reasonable (notably here,
> independent of Python implementation (py rather than cp) but
> architecture specific). I consider this a flawed design, but it's
> always been like this.
> 
> You'll need to raise a bug report for this. Actually two, as the
> pep425tags code is in both pip and wheel independently (the one you
> actually *want* is the pip one, though).

Apparently already reported to pip :-)
https://github.com/pypa/pip/issues/1870
Reported to wheel at
https://bitbucket.org/pypa/wheel/issue/125/get_supported-should-return-more

Thanks for the explanations.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] "not a supported wheel on this platform"

[Antoine Pitrou]

On Tue, 28 Oct 2014 14:44:22 +
Paul Moore  wrote:
> Can you see what the following says (results here are from my Windows
> Python 3.4 system)?
> 
> >>> from wheel.pep425tags import get_supported
> >>> ['-'.join(t) for t in get_supported()]
> ['cp34-none-win_amd64', 'cp34-none-any', 'cp3-none-any',
> 'cp33-none-any', 'cp32-none-any', 'cp31-none-any', 'cp30-none-any',
> 'py34-none-any', 'py3-none-any', 'py33-none-any', 'py32-none-any',
> 'py31-none-any', 'py30-none-any']
> 
> I suspect you might be missing the combination "py2" with an
> architecture other than "any".

Does matching really happen that way?

Here it is anyway. There's no "py2" at all:

>>> ['-'.join(t) for t in get_supported()]  
['cp34-cp34m-linux_x86_64', 'cp34-abi3-linux_x86_64',
'cp34-none-linux_x86_64', 'cp34-none-any', 'cp3-none-any',
'cp33-none-any', 'cp32-none-any', 'cp31-none-any', 'cp30-none-any',
'py34-none-any', 'py3-none-any', 'py33-none-any', 'py32-none-any',
'py31-none-any', 'py30-none-any']

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] "not a supported wheel on this platform"

[Antoine Pitrou]

Hello,

Ok, so until I find a better solution I've tried hosting a wheel on a
personal server, but then I get the following error when installing it:

$ pyvenv-3.4 t
$ source t/bin/activate
(t) $ pip install 
https://ssl.pitrou.net/llvmlite-0.1-py2.py3-none-linux_x86_64.whl
llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this 
platform.
Storing debug log for failure in /home/antoine/.pip/pip.log

Yet:

(t) $ python 
Python 3.4.1 (3.4:d1bf37def4fd, May 19 2014, 19:52:53) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'


The log file doesn't tell anything really interesting:


/home/antoine/t/t/bin/pip run on Tue Oct 28 15:18:06 2014
llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this 
platform.
Exception information:
Traceback (most recent call last):
  File "/home/antoine/t/t/lib/python3.4/site-packages/pip/basecommand.py", line 
122, in main
status = self.run(options, args)
  File "/home/antoine/t/t/lib/python3.4/site-packages/pip/commands/install.py", 
line 257, in run
InstallRequirement.from_line(name, None))
  File "/home/antoine/t/t/lib/python3.4/site-packages/pip/req.py", line 167, in 
from_line
raise UnsupportedWheel("%s is not a supported wheel on this platform." % 
wheel.filename)
pip.exceptions.UnsupportedWheel: llvmlite-0.1-py2.py3-none-linux_x86_64.whl is 
not a supported wheel on this platform.


Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Making a wheel platform-specific?

[Antoine Pitrou]

On Tue, 28 Oct 2014 10:04:17 -0400
Donald Stufft  wrote:
> 
> > On Oct 28, 2014, at 9:43 AM, Antoine Pitrou  wrote:
> > 
> >> I think twine can do that for you (and is generally recommended these
> >> days over setup.py upload, as it uses https).
> > 
> > setup.py upload also uses https these days, AFAIK.
> 
> Paul forgot an important word there, *verified* HTTPS.

Yes, you're right. Thanks for doing this.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig