Re: [Pythonmac-SIG] Problem with pip

2018-01-18 Thread Christopher Barker 
This would be a very good time to go over to Python-dev and see what’s
being done for 3.7.

And to help Ned if you want to see changes.

-CHB

On Thu, Jan 18, 2018 at 10:19 AM Joni Orponen 
wrote:

> On Thu, Jan 18, 2018 at 5:05 PM, Karsten Wolf via Pythonmac-SIG <
> pythonmac-sig@python.org> wrote:
>
>>
>> > On 18.01.2018, at 14:58, Joni Orponen  wrote:
>> >
>> > The offical CPython distributions seem to be compiled on 10.6 for
>> compatibility reasons. It is easier to compile CPython forwards-compatible
>> than to manually patch pyconfig.h after configure to exclude the syscalls
>> not available on your older target platforms.
>>
>> With the following config line you compile a backwards compatible Python
>> (here 64-bit only since I don't like fiddling around with hard to compile
>> fat-libs).
>>
>> # for 2.7
>> env MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch x86_64 " LDFLAGS="-arch
>> x86_64 " ./configure --enable-framework --enable-ipv6 --enable-toolbox-glue
>>
>> and it yields a Python that can be py2apped and run on 10.6 as long as
>> you don't use newer Frameworks or libs that need OSX > 1.6
>>
>
> I'm not sharing your experience.
>
> Adding -Wl,-no_weak_imports to LDFLAGS flags things up and those then
> actually fail hard at runtime. The easiest one to spot and verify yourself
> is getentropy() when using TLS. I'm having to patch out the detected
> support for those syscalls manually out of the generated pyconfig.h when
> compiling something backwards-compatibly for distribution.
>
> I'm also passing through -mmacosx-version-min in both LDFLAGS and CFLAGS
> as not all egg specific build systems pass in the environment correctly,
> but there is still some hope of them to use whatever it is that Python was
> built with as Python exposes that at runtime. There are eggs one still has
> to patch up manually for distribution. This is also useful to be in the
> know of for some of the dependencies of any CPython modules you might have
> to bundle as static libraries for distribution purposes.
>
> For the framework incompatibilities you can get to about the same
> detection ability if you use -Wunguarded-availability in CFLAGS. There is
> also -Wno-unguarded-availability-new which combines to provide the ability
> for not yet caring of non-fatal issues for the newest one (as in the one
> you are currently building on with the intention of being
> backwards-compatible only), but this is rather new [1].
>
> I do not recommend -Werror and -Wunguarded-availability together when
> compiling backwards-compatible as you'd have to fix the use of all the
> deprecated, but not yet broken, bits in CPython itself.
>
> So for targeting 10.6 I'd use something roughly like (also allowing you to
> switch between Xcode versions via xcode-select, if needed):
>
> CC="xcrun cc"
> CXX="$CC"
>
> MACOSX_DEPLOYMENT_TARGET="10.6"
>
> XCODE_ROOT="$(xcode-select --print-path)"
> SYSROOT="$XCODE_ROOT/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"
>
> COMMON_FLAGS="--sysroot=$SYSROOT
> -mmacosx-version-min=$MACOSX_DEPLOYMENT_TARGET"
>
> CFLAGS="-I$SYSROOT/usr/include $COMMON_FLAGS
> -Wno-unguarded-availability-new -Wunguarded-availability"
> CPPFLAGS="$CFLAGS"
>
> LDFLAGS="-L$SYSROOT/usr/lib $COMMON_FLAGS -Wl,-no_weak_imports"
>
> In practice I'd not target such an old OS version when building
> backwards-compatibly as the associated workload would be rather large.
> Building on the latest version and supporting a couple of versions back
> down is still rather manageable and sane in my experience.
>
> Against this backdrop, I'm understanding of people, especially open source
> volunteers, choosing to build both CPython and their distributable wheels
> on 10.6 as the amount of effort to validate and support more permutations
> is rather large for almost no benefit.
>
> The expectation levels on the Windows side have risen and thus since
> Python 3.5 that stack has moved to build with a newer Visual Studio and is
> also offering a nicer userspace install. I'm taking this thread as a signal
> of the expectations of the macOS crowd having risen as well.
>
> A midway compromise point might be to coerce the community to start
> building on 10.11 as 10.12 was again a great hardware support divider such
> as 10.7 was and it's starting to have been long enough since 10.11.
>
> On the actual topic: If one explicitly needs some build-time details to be
> specific, one can just build the egg oneself. Otherwise whatever pip pulls
> in and is working is good enough for the job.
>
> --
> Joni Orponen
>
> [1] https://reviews.llvm.org/D34264
> ___
> Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
> https://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG
>
-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
__

Re: [Pythonmac-SIG] Problem with pip

2018-01-18 Thread Joni Orponen 
On Thu, Jan 18, 2018 at 5:05 PM, Karsten Wolf via Pythonmac-SIG <
pythonmac-sig@python.org> wrote:

>
> > On 18.01.2018, at 14:58, Joni Orponen  wrote:
> >
> > The offical CPython distributions seem to be compiled on 10.6 for
> compatibility reasons. It is easier to compile CPython forwards-compatible
> than to manually patch pyconfig.h after configure to exclude the syscalls
> not available on your older target platforms.
>
> With the following config line you compile a backwards compatible Python
> (here 64-bit only since I don't like fiddling around with hard to compile
> fat-libs).
>
> # for 2.7
> env MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch x86_64 " LDFLAGS="-arch
> x86_64 " ./configure --enable-framework --enable-ipv6 --enable-toolbox-glue
>
> and it yields a Python that can be py2apped and run on 10.6 as long as you
> don't use newer Frameworks or libs that need OSX > 1.6
>

I'm not sharing your experience.

Adding -Wl,-no_weak_imports to LDFLAGS flags things up and those then
actually fail hard at runtime. The easiest one to spot and verify yourself
is getentropy() when using TLS. I'm having to patch out the detected
support for those syscalls manually out of the generated pyconfig.h when
compiling something backwards-compatibly for distribution.

I'm also passing through -mmacosx-version-min in both LDFLAGS and CFLAGS as
not all egg specific build systems pass in the environment correctly, but
there is still some hope of them to use whatever it is that Python was
built with as Python exposes that at runtime. There are eggs one still has
to patch up manually for distribution. This is also useful to be in the
know of for some of the dependencies of any CPython modules you might have
to bundle as static libraries for distribution purposes.

For the framework incompatibilities you can get to about the same detection
ability if you use -Wunguarded-availability in CFLAGS. There is
also -Wno-unguarded-availability-new which combines to provide the ability
for not yet caring of non-fatal issues for the newest one (as in the one
you are currently building on with the intention of being
backwards-compatible only), but this is rather new [1].

I do not recommend -Werror and -Wunguarded-availability together when
compiling backwards-compatible as you'd have to fix the use of all the
deprecated, but not yet broken, bits in CPython itself.

So for targeting 10.6 I'd use something roughly like (also allowing you to
switch between Xcode versions via xcode-select, if needed):

CC="xcrun cc"
CXX="$CC"

MACOSX_DEPLOYMENT_TARGET="10.6"

XCODE_ROOT="$(xcode-select --print-path)"
SYSROOT="$XCODE_ROOT/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"

COMMON_FLAGS="--sysroot=$SYSROOT
-mmacosx-version-min=$MACOSX_DEPLOYMENT_TARGET"

CFLAGS="-I$SYSROOT/usr/include $COMMON_FLAGS
-Wno-unguarded-availability-new -Wunguarded-availability"
CPPFLAGS="$CFLAGS"

LDFLAGS="-L$SYSROOT/usr/lib $COMMON_FLAGS -Wl,-no_weak_imports"

In practice I'd not target such an old OS version when building
backwards-compatibly as the associated workload would be rather large.
Building on the latest version and supporting a couple of versions back
down is still rather manageable and sane in my experience.

Against this backdrop, I'm understanding of people, especially open source
volunteers, choosing to build both CPython and their distributable wheels
on 10.6 as the amount of effort to validate and support more permutations
is rather large for almost no benefit.

The expectation levels on the Windows side have risen and thus since Python
3.5 that stack has moved to build with a newer Visual Studio and is also
offering a nicer userspace install. I'm taking this thread as a signal of
the expectations of the macOS crowd having risen as well.

A midway compromise point might be to coerce the community to start
building on 10.11 as 10.12 was again a great hardware support divider such
as 10.7 was and it's starting to have been long enough since 10.11.

On the actual topic: If one explicitly needs some build-time details to be
specific, one can just build the egg oneself. Otherwise whatever pip pulls
in and is working is good enough for the job.

-- 
Joni Orponen

[1] https://reviews.llvm.org/D34264
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] Problem with pip

2018-01-18 Thread Karsten Wolf via Pythonmac-SIG 

> On 18.01.2018, at 14:58, Joni Orponen  wrote:
> 
> The offical CPython distributions seem to be compiled on 10.6 for 
> compatibility reasons. It is easier to compile CPython forwards-compatible 
> than to manually patch pyconfig.h after configure to exclude the syscalls not 
> available on your older target platforms.

With the following config line you compile a backwards compatible Python (here 
64-bit only since I don't like fiddling around with hard to compile fat-libs).

# for 2.7
env MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch x86_64 " LDFLAGS="-arch x86_64 
" ./configure --enable-framework --enable-ipv6 --enable-toolbox-glue

and it yields a Python that can be py2apped and run on 10.6 as long as you 
don't use newer Frameworks or libs that need OSX > 1.6


-karsten

___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

Re: [Pythonmac-SIG] Problem with pip

2018-01-18 Thread Joni Orponen 
On Wed, Jan 17, 2018 at 6:42 PM, Shawn Tolidano 
wrote:
>
> So - to my knowledge, it has NO BUSINESS using the 10.6 as a compatible
> wheel and should instead download the source.
>

The offical CPython distributions seem to be compiled on 10.6 for
compatibility reasons. It is easier to compile CPython forwards-compatible
than to manually patch pyconfig.h after configure to exclude the syscalls
not available on your older target platforms.

For the people compiling wheels for distribution it would make sense to
mirror CPython for their distributions for similar reasons.

Is there some newer syscall path you absolutely need to go through (for
reasons of sandboxing requirements? a specification requirement for using
getentropy()?) or what is the actual problem you are trying to fix here?

-- 
Joni Orponen
___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

[Pythonmac-SIG] Problem with pip

2018-01-17 Thread Shawn Tolidano 
Hi,

I have cffi==1.11.4 in my requirements file (and in my transitive file, by 
extension).
I am running High Sierra, my python is 2.7.14 and comes from Brew.

/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/

I create a venv and upgrade pip (to 9) and then:
./build-support/python/venvs/pydeps-wheel/bin/pip install -r 
3rdparty/python/requirements.transitive.txt

In this file are lots of requirements, but that’s not relevant (I don’t think). 
One of them is cffi==1.11.4.

When I run this, this happens:
Collecting cffi==1.11.4 (from -r 3rdparty/python/requirements.transitive.txt 
(line 22))
  Downloading cffi-1.11.4-cp27-cp27m-macosx_10_6_intel.whl (238kB)
100% || 245kB 2.2MB/s

Note the 10.6.

Also note this:
./build-support/python/venvs/pydeps-wheel/bin/python -c "import distutils.util; 
print(distutils.util.get_platform())"
macosx-10.13-x86_64

So - to my knowledge, it has NO BUSINESS using the 10.6 as a compatible wheel 
and should instead download the source.

So I empty my requirements file and leave ONLY cffi==1.11.4
I recompute (with pip freeze) and end up with cffi, setuptools, wheel, and 
pycparser.
I run the SAME install command as above.

And get this:
Collecting cffi==1.11.4 (from -r 3rdparty/python/requirements.transitive.txt 
(line 2))
  Using cached cffi-1.11.4.tar.gz

And ultimately, it builds this:
Dist/wheels/cffi-1.11.4-cp27-none-macosx_10_13_x86_64.whl

1) Why does it think 10.6 is a valid wheel?
2) How do I fix this?

Thanks a lot in advance for your help!

-
Shawn Tolidano
Senior Engineer - DevOps

90 Fifth Avenue 6th Floor
New York, NY 10011
m: (516) 668-5448

 

NEW YORK | BROOKLYN | EAST HAMPTON | BRIDGEHAMPTON | SOUTHAMPTON | SAG HARBOR | 
WASHINGTON DC | CHEVY CHASE | BOSTON | CAMBRIDGE | MIAMI | COCONUT GROVE | 
BEVERLY HILLS | MALIBU | PASADENA | MONTECITO | SANTA BARBARA | BASALT | ASPEN 
| SAN FRANCISCO


___
Pythonmac-SIG maillist  -  Pythonmac-SIG@python.org
https://mail.python.org/mailman/listinfo/pythonmac-sig
unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG