Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthew Brett]

On Sun, Jan 31, 2016 at 11:25 PM, Marius Gedminas  wrote:
> On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote:
>> Hi,
>>
>> > I can confirm that Debian and Anaconda builds of CPython 2.7 both have
>> > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
>> > 0x. Hmm. I guess they should fix that.
>> >
>> > Also the manylinux docker image currently has sys.maxunicode ==
>> > 0x, so we should definitely fix that :-).
>>
>> A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
>> unicode by default.  Are there any known distributions providing UCS2
>> unicode Pythons?
>
> I don't know of any.

I also tested on Debian wheezy 32-bit; Fedora 22 (32-bit packages);
openSUSE 13.2, recent Arch and Gentoo, these are all wide unicode.

> Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow
> unicode builds.

Ouch - good to know.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthias Klose]

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python in 
Debian/Ubuntu.  So I would like to have some feedback from maintainers of Python 
in other Linux distributions (Nick, no, you're not one of these).


The proposal just takes some environment and declares that as a standard.  So 
everybody wanting to supply these wheels basically has to use this environment. 
Without giving any details, without giving any advise how to produce such wheels 
in other environments. Without giving any hints how such wheels may be broken 
with newer environments.  Without mentioning this is am64/i386 only.
There might be more. Pretty please be specific about your environment.  Have a 
look how the LSB specifies requirements on the runtime environment ... and then 
ask yourself why the lsb doesn't have any real value.


Matthias

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Donald Stufft]

> On Feb 1, 2016, at 6:37 PM, Matthias Klose  wrote:
> 
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>> Hi all,
>> 
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
> 
> I don't think so.  I am biased because I'm the maintainer for Python in 
> Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
> Python in other Linux distributions (Nick, no, you're not one of these).
> 
> The proposal just takes some environment and declares that as a standard.  So 
> everybody wanting to supply these wheels basically has to use this 
> environment. Without giving any details, without giving any advise how to 
> produce such wheels in other environments. Without giving any hints how such 
> wheels may be broken with newer environments.

I’m not sure this is true. It tells you exactly what versions of glibc and 
other libraries it is allowed to link against. It can link against older if it 
wants, it can’t link against newer.

> Without mentioning this is am64/i386 only.

First sentence: This PEP proposes the creation of a new platform tag for Python 
package built distributions, such as wheels, calledmanylinux1_{x86_64,i686} 
with external dependencies limited to a standardized, restricted subset of the 
Linux kernel and core userspace ABI.

Later on: Because CentOS 5 is only available for x86_64 and i686 architectures, 
these are the only architectures currently supported by the manylinux1 policy.

I think it’s a reasonable policy too, AMD64 is responsible for an order of 
magnitude more downloads than all other architectures on Linux combined 
(71,424,040 vs 1,086,527 in current data set). If you compare AMD64+i386 
against everything else then you’re looking at two orders of magnitude 
(72,142,511 vs 368,056). I think we can live with a solution that covers 99.5% 
of all Linux downloads from PyPI.

> There might be more. Pretty please be specific about your environment.  Have 
> a look how the LSB specifies requirements on the runtime environment ... and 
> then ask yourself why the lsb doesn't have any real value.
> 

Instead of vague references to the LSB, can you tell us why you think the LSB 
doesn’t have any real value, and importantly, how that relates to trying to 
determine a minimum set of binary ABI. In addition, can you clarify why, if 
your assertion is this isn’t going to work, can you state why it won’t work 
when it is working for many user’s in the wild through Anaconda, Enthought, the 
Holy Build Box, etc?


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nathaniel Smith]

On Mon, Feb 1, 2016 at 3:37 PM, Matthias Klose  wrote:
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>
>
> I don't think so.  I am biased because I'm the maintainer for Python in
> Debian/Ubuntu.

Thank you for your work! I've been using Debian/Ubuntu as my sole
desktop and for Python development since ~2000, so it's very much
appreciated :-).

> So I would like to have some feedback from maintainers of
> Python in other Linux distributions (Nick, no, you're not one of these).
>
> The proposal just takes some environment and declares that as a standard.
> So everybody wanting to supply these wheels basically has to use this
> environment.

Yeah, pretty much. The spec says "we define the ABI by reference to
this environment" and if you can find some other ways to build these
wheels without using that environment, then go for it, but in practice
I doubt there's much point. CentOS 5 is widely available, free (in all
senses), and works. Even if it does require one to use weird tools
like "yum" and "rpm" whose UIs are constantly confusing me and make no
sense whatsoever [1].

[1] i.e., they aren't identical to apt / dpkg.

> Without giving any details, without giving any advise how to
> produce such wheels in other environments.

There isn't a lot of end-user-focused documentation in the PEP itself,
it's true, but that's because that's how PEPs work.

Here's an example project for building manylinux1 wheels using
Travis-CI (which runs some version of Ubuntu):

  https://github.com/pypa/python-manylinux-demo

Obviously this is all still a work in progress ("latest commit: 40
minutes ago"). Better docs are coming :-)

> Without giving any hints how such
> wheels may be broken with newer environments.

What kind of hints are you suggesting?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Alexander Walters]

On 2/1/2016 18:37, Matthias Klose wrote:

On 30.01.2016 00:29, Nathaniel Smith wrote:

Hi all,

I think this is ready for pronouncement now -- thanks to everyone for
all their feedback over the last few weeks!


I don't think so.  I am biased because I'm the maintainer for Python 
in Debian/Ubuntu.  So I would like to have some feedback from 
maintainers of Python in other Linux distributions (Nick, no, you're 
not one of these).


The proposal just takes some environment and declares that as a 
standard.  So everybody wanting to supply these wheels basically has 
to use this environment. Without giving any details, without giving 
any advise how to produce such wheels in other environments. Without 
giving any hints how such wheels may be broken with newer 
environments.  Without mentioning this is am64/i386 only.
There might be more. Pretty please be specific about your 
environment.  Have a look how the LSB specifies requirements on the 
runtime environment ... and then ask yourself why the lsb doesn't have 
any real value.


Matthias

I... Thought the environment this pep describes is the docker image, and 
only the docker image, and anything not made on that docker image is in 
violation of the pep.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Robert T. McGibbon]

On Mon, Feb 1, 2016 at 3:47 PM, Alexander Walters 
wrote:

>
>
> On 2/1/2016 18:37, Matthias Klose wrote:
>
>> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>
>>> Hi all,
>>>
>>> I think this is ready for pronouncement now -- thanks to everyone for
>>> all their feedback over the last few weeks!
>>>
>>
>> I don't think so.  I am biased because I'm the maintainer for Python in
>> Debian/Ubuntu.  So I would like to have some feedback from maintainers of
>> Python in other Linux distributions (Nick, no, you're not one of these).
>>
>> The proposal just takes some environment and declares that as a
>> standard.  So everybody wanting to supply these wheels basically has to use
>> this environment. Without giving any details, without giving any advise how
>> to produce such wheels in other environments. Without giving any hints how
>> such wheels may be broken with newer environments.  Without mentioning this
>> is am64/i386 only.
>> There might be more. Pretty please be specific about your environment.
>> Have a look how the LSB specifies requirements on the runtime environment
>> ... and then ask yourself why the lsb doesn't have any real value.
>>
>> Matthias
>>
>> I... Thought the environment this pep describes is the docker image, and
> only the docker image, and anything not made on that docker image is in
> violation of the pep.



It's not correct that anything made outside the docker image is in
violation of the PEP. The docker images are just tools that can help you
compile compliant wheels. Nathaniel and I tried to describe this as
precisely as we could. See this section
 of the PEP.

To comply with the policy, the wheel needs to (a not link against any other
external libraries beyond those mentioned in the PEP, (b) *work* on a stock
CentOS 5.11 machine, and (c) not use any narrow-unicode symbols (only
relevant for Python < 3.2). A consequence of requirements (a) and (b) is
that versioned symbols that are referenced in the depended-upon shared
libraries need to use sufficiently old versions of the symbols, which are
noted in the PEP as well. In order to satisfy this aspect of the policy,
the easiest route, from our experience, is to compile the wheel inside a
CentOS 5 environment, but other routes are possible, including

 - statically link everything
 - use your favorite linux distro, but install an older version of glibc
and configure your compiler to point to that.
 - use some inline assembly to instruct the linker to prefer older symbol
versions in libraries like glibc.
 - etc.

I also wrote the auditwheel command line tool that can check to see if a
wheel is manylinux1 compatible, and give you some feedback about what to
fix if it's not.

And furthermore, I've just put up an example project on github that you can
use as a template for compiling manylinux1 wheels using Travis-CI. You can
find it here: https://github.com/pypa/python-manylinux-demo

-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Marius Gedminas]

On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote:
> Hi,
> 
> > I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> > 0x. Hmm. I guess they should fix that.
> >
> > Also the manylinux docker image currently has sys.maxunicode ==
> > 0x, so we should definitely fix that :-).
> 
> A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
> unicode by default.  Are there any known distributions providing UCS2
> unicode Pythons?

I don't know of any.

Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow
unicode builds.

Marius Gedminas
-- 
If you are angry with someone, you should walk a mile in their shoes... then
you'll be a mile away from them, and you'll have their shoes.


signature.asc
Description: Digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Nick Coghlan]

On 31 January 2016 at 02:49, Donald Stufft  wrote:
>
>> On Jan 30, 2016, at 3:58 AM, Nick Coghlan  wrote:
>>
>> I also think this version covers everything we need it to cover, so
>> I'm going to mark it as Active and point to this post as the
>> resolution :)
>
> Hilariously I just read on Hacker news:
>
> "Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux 
> and are in the process of switching the Docker official image library from 
> ubuntu to Alpine.” [1]
>
> So we might end up needing a MUSL based platform tag in the near future ;)

Aye, I saw that, too. Alpine is currently the only MUSL based distro
I'm aware of, so it may be enough to ensure that we include Alpine in
the list of distros we ensure are correctly supported by
distro-specific wheel tags.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 1, 2016, at 3:37 PM, Matthias Klose  wrote:
> 
> On 30.01.2016 00:29, Nathaniel Smith wrote:
>> Hi all,
>> 
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
> 
> I don't think so.  I am biased because I'm the maintainer for Python in 
> Debian/Ubuntu.  So I would like to have some feedback from maintainers of 
> Python in other Linux distributions (Nick, no, you're not one of these).

Possibly, but it would be very helpful for such maintainers to limit their 
critique to "in what scenarios will this fail for users" and not have the whole 
peanut gallery chiming in with "well on _my_ platform we would have done it 
_this_ way".

I respect what you've done for Debian and Ubuntu, Matthias, and I use the heck 
out of that work, but honestly this whole message just comes across as sour 
grapes that someone didn't pick a super-old Debian instead of a super-old Red 
Hat.  I don't think it's promoting any progress.

> The proposal just takes some environment and declares that as a standard.  So 
> everybody wanting to supply these wheels basically has to use this 
> environment.

There's already been lots of discussion about how this environment is a lowest 
common denominator.  Many other similar environments could _also_ be lowest 
common denominator.

> Without giving any details, without giving any advise how to produce such 
> wheels in other environments. Without giving any hints how such wheels may be 
> broken with newer environments.

They won't be.  That's the whole point.

> Without mentioning this is am64/i386 only.

Wheels already have an architecture tag, separate from the platform tag, so 
this being "am64/i386" is irrelevant.

> There might be more. Pretty please be specific about your environment.  Have 
> a look how the LSB specifies requirements on the runtime environment ... and 
> then ask yourself why the lsb doesn't have any real value.


In the future, more specific and featureful distro tags sound like a good idea. 
 But could we please stop making the default position on distutils-sig "this 
doesn't cater to my one specific environment in the most optimal possible way, 
so let's give up on progress entirely"?  This is a good proposal that addresses 
environment portability and gives Python a substantially better build-artifact 
story than it currently has, in the environment most desperately needing one 
(server-side linux).  Could it be better?  Of course.  It could be lots better. 
 There are lots of use-cases for dynamically linked wheels and fancy new 
platform library features in newer linuxes.  But that can all come later, and 
none of it needs to have an impact on this specific proposal, right now.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[David Cournapeau]

On Sat, Jan 30, 2016 at 8:37 AM, Nathaniel Smith  wrote:

> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> > On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> >> Hi all,
> >>
> >> I think this is ready for pronouncement now -- thanks to everyone for
> >> all their feedback over the last few weeks!
> >>
> >> The only change relative to the last posting is that we rewrote the
> >> section on "Platform detection for installers", to switch to letting
> >> distributors explicitly control manylinux1 compatibility by means of a
> >> _manylinux module.
> >
> > In terms of the proposal itself, I think this version is excellent :)
> >
> > However, I realised that there's an implicit assumption we've been
> > making that really should be spelled out explicitly: manylinux1 wheels
> > targeting CPython 3.2 and earlier need to be compiled against a
> > CPython built in wide Unicode mode, and in those cases, the detection
> > of manylinux1 compatibility at the platform level should include
> > checking for "sys.maxunicode > 0x".
>
> Doh, excellent catch!
>
> I've just pushed the obvious update to handle this directly to the
> copy of the PEP in the manylinux repository.
>
> Diff:
> https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc
>
> New text in full:
>
> https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst
>
> I haven't sent to the PEP editors, because they already have another
> diff from me sitting in their inboxes and I'm not sure how to do this
> in a way that doesn't confuse things :-)
>
> > The main reason we need to spell this out explicitly is that while
> > distros (and I believe other redistributors) build CPython-for-Linux
> > in wide mode as a matter of course, a Linux checkout of CPython 2.7
> > will build in narrow mode by default.
>
> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>

Yes, they should :) I am not sure why it was built this way (before my
time), it is unfortunately not easy to fix when you have a large existing
customer base.

David


> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nate Coraor]

On Fri, Jan 29, 2016 at 11:44 PM, Donald Stufft  wrote:

>
> On Jan 29, 2016, at 2:35 PM, Nate Coraor  wrote:
>
> Is there a distro-specific wheel tagging PEP in development somewhere that
> I missed? If not, I will get the ball rolling on it.
>
>
>
> I think this a great idea, and I think it actually pairs nicely with the
> manylinux proposal. It should be pretty easy to cover the vast bulk of
> users with a handful of platform specific wheels (1-3ish) and then a
> manylinux wheel to cover the rest. It would let a project use newer
> toolchains/libraries in the common case, but still fall back to the older
> ones on more unusual platforms.
>

Fantastic, this is exactly the sort of usage I was hoping to see. I'll move
forward with it, then.

--nate


>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Nate Coraor]

On Fri, Jan 29, 2016 at 9:14 PM, Nick Coghlan  wrote:

> On 30 January 2016 at 05:30, Nate Coraor  wrote:
> > I wonder if, in relation to this, it may be best to have two separate
> tags:
> > one to indicate that the wheel includes external libraries rolled in to
> it,
> > and one to indicate that it doesn't. That way, a user can make a
> conscious
> > decision as to whether they want to install any wheels that could include
> > libraries that won't be maintained by the distribution package manager.
> That
> > way if we end up in a future world where manylinux wheels and
> > distro-specific wheels (that may depend on non-default distro packages)
> live
> > in PyPI together, there'd be a way to indicate a preference.
>
> I don't think we want to go into that level of detail in the platform
> tag, but metadata for bundled pre-built binaries in wheels and
> vendored dependencies in sdists is worth considering as an enhancement
> in its own right.
>

I thought the same thing - the only reason I proposed tags that it was my
understanding that such metadata is not available to installation tool(s)
until the distribution is fetched and inspected. If my limited
understanding is incorrect then I agree that having this in the tags is too
much.

--nate


>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig