Re: [Distutils] draft PEP: manylinux1

2016-01-30 Thread Glyph Lefkowitz 

> On Jan 29, 2016, at 8:44 PM, Donald Stufft  wrote:
> 
> 
>> On Jan 29, 2016, at 2:35 PM, Nate Coraor > > wrote:
>> 
>> Is there a distro-specific wheel tagging PEP in development somewhere that I 
>> missed? If not, I will get the ball rolling on it.
> 
> 
> I think this a great idea, and I think it actually pairs nicely with the 
> manylinux proposal. It should be pretty easy to cover the vast bulk of users 
> with a handful of platform specific wheels (1-3ish) and then a manylinux 
> wheel to cover the rest. It would let a project use newer 
> toolchains/libraries in the common case, but still fall back to the older 
> ones on more unusual platforms.

Yes!  This would be fantastic.  There are some libraries you actually want to 
dynamically link against from the platform, especially if you're writing 
desktop apps.  On OS X you can do this because /System/*/ is more or less fixed 
when you are >= some version; on linux less so but it would be very nice to 
build artifacts for specific versions when possible.

-glyph

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Incorrect detection of 32/64-bit arch on Linux for old 32-bit VMs migrated to 64-bit kernels

2016-01-30 Thread Nathaniel Smith 
Here's an interesting bug I just discovered on the old and poorly
maintained VM that I keep for running random junk:

# 64-bit kernel
$ uname -a
Linux roberts 4.1.5-x86_64-linode61 #7 SMP Mon Aug 24 13:46:31 EDT
2015 x86_64 x86_64 x86_64 GNU/Linux

# 32-bit userland
$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
BuildID[sha1]=0xf481c2b1f8b4328b2f56b642022cc35b4ad91b61, stripped

$ /usr/bin/python2.7

# Yep, definitely 32-bit
>>> import sys; sys.maxint
2147483647

# Uh oh.
>>> import distutils.util; distutils.util.get_platform()
'linux-x86_64'

# Yeah, this is bad
>>> import distlib.wheel; distlib.wheel.COMPATIBLE_TAGS
set([(u'cp21', u'none', u'any'), (u'py21', u'none', u'any'), (u'py23',
u'none', u'any'), (u'cp24', u'none', u'any'), (u'py2', u'none',
u'any'), (u'cp27', u'none', u'any'), (u'cp20', u'none', u'any'),
(u'py22', u'none', u'any'), (u'cp26', u'none', u'any'), (u'cp27',
u'cp27mu', u'linux_x86_64'), (u'py25', u'none', u'any'), (u'cp23',
u'none', u'any'), (u'py24', u'none', u'any'), (u'cp25', u'none',
u'any'), (u'py27', u'none', u'any'), (u'cp22', u'none', u'any'),
(u'cp2', u'none', u'any'), (u'cp27', u'none', u'linux_x86_64'),
(u'py26', u'none', u'any'), (u'py20', u'none', u'any')])

In the past this has never mattered, because there were no linux
wheels on pypi, so even if pip was using the wrong platform tag it
still wouldn't download the wrong thing. But once manylinux1 goes
live, any systems configured like this will start downloading and
installing totally wheels that will crash on import.

The problem seems to be that distutils.util.get_platform() assumes
that the architecture is whatever uname's "machine" field says
(equivalent to uname -m). Unfortunately, this gives the architecture
of the kernel, not of the Python interpreter.

I think the fix is that we should add some check like

if osname == "linux" and machine == "x86_64" and sys.maxsize == 2147483647:
machine = "i686"

to distutils (in the branches where it's still maintained), and also
need to add a similar workaround to distlib? (With the later needing
to land before or together with manylinux1 enablement.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Nathaniel Smith 
On Sat, Jan 30, 2016 at 12:37 AM, Nathaniel Smith  wrote:
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).

This is fixed now (or will be in a few minutes when this build
finishes: 
https://quay.io/repository/manylinux/manylinux/build/40961d45-aff6-4ed2-a778-f73413340800)

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Nathaniel Smith 
On Sat, Jan 30, 2016 at 12:58 AM, Nick Coghlan  wrote:
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Awesome! Thanks everyone :-)

Of course, as soon as this happened I discovered two small but
substantive bugs in the text...

- the text refers to 32-bit linux as "i386" (because I foolishly
trusted PEP 425); it turns out that on any modern system the platform
tag is "linux_i686".
- there was a small bug in the is_manylinux1_compatible sample code,
because I missed that distutils.util.get_platform() uses a hyphen
separator instead of an underscore. (But now the sample code is
running as part of the test suite for the docker image, so I know it
works.)

Both are fixed in the diff that I just sent to the pep editors:
  https://github.com/manylinux/manylinux/pull/20/files

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] ANN: distlib 0.2.2 released on PyPI

2016-01-30 Thread Vinay Sajip 
I've just released version 0.2.2 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended to be
usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

* Fixed issue #81: Added support for detecting distributions installed by
  wheel versions >= 0.23 (which use metadata.json rather than pydist.json).
* Updated default PyPI URL to https://pypi.python.org/pypi
* Updated to use different formatting for description field for V1.1
  metadata.
* Corrected “classifier” to “classifiers” in the mapping for V1.0 metadata.
* Improved support for Jython when quoting executables in output scripts.
* Fixed issue #77: Made the internal URL used for extended metadata fetches
  configurable via a module attribute.
* Fixed issue #78: Improved entry point parsing to handle leading spaces in
  ini-format files.

A more detailed change log is available at [2].

Please try it out, and if you find any problems or have any suggestions for
improvements, please give some feedback using the issue tracker! [3]

Regards,

Vinay Sajip

[1] https://pypi.python.org/pypi/distlib/0.2.2
[2] https://goo.gl/M3kQzR
[3] https://bitbucket.org/pypa/distlib/issues/new
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Donald Stufft 

> On Jan 30, 2016, at 3:58 AM, Nick Coghlan  wrote:
> 
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

Hilariously I just read on Hacker news:

"Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux 
and are in the process of switching the Docker official image library from 
ubuntu to Alpine.” [1]

So we might end up needing a MUSL based platform tag in the near future ;)


[1] https://news.ycombinator.com/item?id=11000827

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Nathaniel Smith 
On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan  wrote:
> On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
>> Hi all,
>>
>> I think this is ready for pronouncement now -- thanks to everyone for
>> all their feedback over the last few weeks!
>>
>> The only change relative to the last posting is that we rewrote the
>> section on "Platform detection for installers", to switch to letting
>> distributors explicitly control manylinux1 compatibility by means of a
>> _manylinux module.
>
> In terms of the proposal itself, I think this version is excellent :)
>
> However, I realised that there's an implicit assumption we've been
> making that really should be spelled out explicitly: manylinux1 wheels
> targeting CPython 3.2 and earlier need to be compiled against a
> CPython built in wide Unicode mode, and in those cases, the detection
> of manylinux1 compatibility at the platform level should include
> checking for "sys.maxunicode > 0x".

Doh, excellent catch!

I've just pushed the obvious update to handle this directly to the
copy of the PEP in the manylinux repository.

Diff: 
https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc

New text in full:
https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst

I haven't sent to the PEP editors, because they already have another
diff from me sitting in their inboxes and I'm not sure how to do this
in a way that doesn't confuse things :-)

> The main reason we need to spell this out explicitly is that while
> distros (and I believe other redistributors) build CPython-for-Linux
> in wide mode as a matter of course, a Linux checkout of CPython 2.7
> will build in narrow mode by default.

I can confirm that Debian and Anaconda builds of CPython 2.7 both have
sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
0x. Hmm. I guess they should fix that.

Also the manylinux docker image currently has sys.maxunicode ==
0x, so we should definitely fix that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Robert T. McGibbon 
Oh yes, this is very important! I will have to put a check in auditwheel as
well to verify this in the extensions too.

-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Oscar Benjamin 
On 30 January 2016 at 08:58, Nick Coghlan  wrote:
>
> I applied both this iteration and the previous one to the PEPs repo in
> order to review it, so modulo caching issues, this latest draft is
> live now.
>
> I also think this version covers everything we need it to cover, so
> I'm going to mark it as Active and point to this post as the
> resolution :)

I had to see PEP 1 to understand what "Active" means but now I see
that it means that this PEP is approved but subject to indefinite
tinkering:
https://www.python.org/dev/peps/pep-0001/#pep-review-resolution

Brilliant, good work everyone! I'm looking forward to this.

So AFAICT the actions are:
1) PyPI allows uploading wheels with manylinux tag.
2) pip updated to recognise these wheels on the appropriate Linux systems.
3) Packagers make and upload wheels

Is there also a policy of informing python-dev once a PEP is approved
here? PEP 1 is ambiguous on that point but something like this really
needs to be known about more widely than this list.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Nick Coghlan 
On 30 January 2016 at 09:29, Nathaniel Smith  wrote:
> Hi all,
>
> I think this is ready for pronouncement now -- thanks to everyone for
> all their feedback over the last few weeks!
>
> The only change relative to the last posting is that we rewrote the
> section on "Platform detection for installers", to switch to letting
> distributors explicitly control manylinux1 compatibility by means of a
> _manylinux module.

In terms of the proposal itself, I think this version is excellent :)

However, I realised that there's an implicit assumption we've been
making that really should be spelled out explicitly: manylinux1 wheels
targeting CPython 3.2 and earlier need to be compiled against a
CPython built in wide Unicode mode, and in those cases, the detection
of manylinux1 compatibility at the platform level should include
checking for "sys.maxunicode > 0x".

The main reason we need to spell this out explicitly is that while
distros (and I believe other redistributors) build CPython-for-Linux
in wide mode as a matter of course, a Linux checkout of CPython 2.7
will build in narrow mode by default.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

2016-01-30 Thread Matthew Brett 
Hi,

> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).

A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
unicode by default.  Are there any known distributions providing UCS2
unicode Pythons?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Incorrect detection of 32/64-bit arch on Linux for old 32-bit VMs migrated to 64-bit kernels

2016-01-30 Thread Greg Ewing 

Nathaniel Smith wrote:

The problem seems to be that distutils.util.get_platform() assumes
that the architecture is whatever uname's "machine" field says
(equivalent to uname -m). Unfortunately, this gives the architecture
of the kernel, not of the Python interpreter.

I think the fix is that we should add some check like

if osname == "linux" and machine == "x86_64" and sys.maxsize == 2147483647:
machine = "i686"


If you fix this, please do it for other platforms as well.
I encountered the same problem on MacOSX, where for historical
reasons I have installed Python and all my compiled-from-source
libraries as 32 bit. The result is that py2app includes the
wrong version of the stub executable in the apps it builds.

Conceivably a similar thing could happen on Windows, although
I haven't tested it.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig