Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Joseph Lee Nunn III]

> On May 09 , 2016, at 11:03 AM, Chris Barker  wrote:
> 
> I installed OSX Python 3.5.1, created a virtual environment using the pyvenv 
> command from that newly installed Python 3.5.1, and sourced its bin/activate.
> Then when running pip install ZODB, the compiler would be passed the 
> -isysroot flag for the 10.6 SDK, which would change its include paths and it 
> would find no includes or libs 
> 
> this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the 
> includes and libs  needed?
> 

That is the weird part, if installing into the system python the new sysroot 
worked just fine.  It was only when trying to install into a virtual 
environment that it failed.

What the difference is I just don’t know.

Thanks again for you efforts!

Joseph___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Chris Barker]

Thanks Joseph,

I say this isn't worth pursuing until someone else reports the issue, as I
couldn't replicate it with a "clean" system.

I had the old 10.6 SDK in /Developer/SDKs, it could have been ‘special’ in
> that I did alot of weird things for a former project and I just don’t
> recall.
>

I suspect your 10.6 SDK was cnfigured oddly, or 

For the sake of explanation though lets say it wasn’t, what then?
>
> I installed OSX Python 3.5.1, created a virtual environment using the
> pyvenv command from that newly installed Python 3.5.1, and sourced its
> bin/activate.
> Then when running pip install ZODB, the compiler would be passed the
> -isysroot flag for the 10.6 SDK, which would change its include paths and
> it would find no includes or libs
>

this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all
the includes and libs  needed?

Not sure why that is, or what would make the virtual environment different,
> but there you go.
>

yeah, me neither 

Anyway, unless someone else sees this, I think we're done.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Joseph Lee Nunn III]

Thank you for looking into this for me, but since I fixed my problem by simply 
deleting the old SDKs I’ve moved on.

However let me clarify exactly what was going on one more time, just in case 
you guys decide to pursue further.

I had the old 10.6 SDK in /Developer/SDKs, it could have been ‘special’ in that 
I did alot of weird things for a former project and I just don’t recall.

For the sake of explanation though lets say it wasn’t, what then?

I installed OSX Python 3.5.1, created a virtual environment using the pyvenv 
command from that newly installed Python 3.5.1, and sourced its bin/activate.
Then when running pip install ZODB, the compiler would be passed the -isysroot 
flag for the 10.6 SDK, which would change its include paths and it would find 
no includes or libs (if you sym link the includes into the include directory of 
the virtual env you will get farther and see that the linker does not find libs 
either).  By suppling adding -v to CFLAGS you can have the compiler output the 
include paths and compiler call when running pip install.  That is how I 
noticed the alternate system root supplied by the isysroot flag.  Grepping 
around found me the _sysconfigdata.py file where that stuff was specified, it 
took a bit to realize that file came with the distribution and wasn’t generated 
on my system.

When not in a virtual environment, ZODB would install just fine into the system 
python (2.7).  

Not sure why that is, or what would make the virtual environment different, but 
there you go.

Joseph Nunn
josephn...@gmail.com
joseph.n...@uci.edu


> On May 07 , 2016, at 11:52 PM, Chris Barker  wrote:
> 
> On Sat, May 7, 2016 at 10:59 PM, Chris Barker  > wrote:
> 
>  Trying to decide if I want to go through the hassle of installing an old 
> XCode to see if i can replicate the bug the OP is reporting
> 
> OK, I found this gitHub repo with all the old SDKs:
> 
> https://github.com/phracker/MacOSX-SDKs 
> 
> 
> I put a copy of the 10.6 SDk in it's old location:
> 
> /Developer/SDKs/MacOSX10.6.sdk/
> 
> /before I did that, I did a pip install of zodb in a virtualenv, and all 
> seemed to go fine. THe extensions rae linked against :
> 
> $ otool -L cPersistence.cpython-35m-darwin.so 
>  
> cPersistence.cpython-35m-darwin.so 
> :
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
> 1197.1.1)
> 
> where that comes from -- who knows?? 
> 
> then I tried again, after putting the sdk in place:
> 
> [note: serious pain the $%^# to get pip to stop using it's cache! ]
> 
> and everything seemed to build fine, and at least one simple test of zodb 
> works fine. And this is how the extensions are linked:
> 
> $  otool -L cPersistence.cpython-35m-darwin.so 
> 
> cPersistence.cpython-35m-darwin.so 
> :
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 
> 125.2.11)
> 
> which is the same as above, but the "current version" is much older -- so 
> does that mean it used the 10.6 SDK
> 
> Anyway, I can't reproduce the OP's bug.
> 
> -CHB
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
> 
> chris.bar...@noaa.gov 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Chris Barker]

On Sat, May 7, 2016 at 10:59 PM, Chris Barker  wrote:

>
>  Trying to decide if I want to go through the hassle of installing an
> old XCode to see if i can replicate the bug the OP is reporting
>

OK, I found this gitHub repo with all the old SDKs:

https://github.com/phracker/MacOSX-SDKs

I put a copy of the 10.6 SDk in it's old location:

/Developer/SDKs/MacOSX10.6.sdk/

/before I did that, I did a pip install of zodb in a virtualenv, and all
seemed to go fine. THe extensions rae linked against :

$ otool -L cPersistence.cpython-35m-darwin.so
cPersistence.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1197.1.1)

where that comes from -- who knows??

then I tried again, after putting the sdk in place:

[note: serious pain the $%^# to get pip to stop using it's cache! ]

and everything seemed to build fine, and at least one simple test of zodb
works fine. And this is how the extensions are linked:

$  otool -L cPersistence.cpython-35m-darwin.so
cPersistence.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.11)

which is the same as above, but the "current version" is much older -- so
does that mean it used the 10.6 SDK

Anyway, I can't reproduce the OP's bug.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Chris Barker]

On Fri, May 6, 2016 at 10:43 AM, Joseph Lee Nunn III 
wrote:

> Then I have found the right list to post on, for my problem is a bug in
> the Python 3.5.1 distribution.  Albeit one which will only manifest in
> certain situations.
>
> The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk`
> directives in the following file
>
>
> /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py
>

As I understand it, that is all there so that when extensions are built,
they are linked against that SDK -- so you can build extensions that can
then be run on any system 10.6 and greater.


> Will cause errors for people who have kept around the old /Developer
> directory tree, and old SDKs like 10.6, for building backwards compatible
> old code.


what are the errors -- wouldn't the old SDK be found, and used, and that's
exactly what's intended?


>   Newer versions of the SDKs are now kept inside the Xcode application
> itself.  And then they will only have a problem when trying to install a
> package with C extensions,


well, sure. of course this is an issue for building C extensions.


> which in my case was Persistence, a dependency of ZODB.  And even more so,
> ONLY when trying to pip install said package into a virtual environment
> created by 3.5’s pyvenv command, if installing the package into the system
> wide Python there will be no error.
>

hmm, As I understand it the building process should be exactly the same
whether you are on a virtualenv or not. but if so -- maybe the "solution"
is to build a wheel outside virtualenv, and then install that wheel in the
virtualenv. But I agree, it should work, of course.


> You see the -isysroot flag has the unusual behavior where instead of
> reporting an error when it cannot find the newly specified system root it
> simply has no effect.  So if you don’t have the SDK specified in the
> location specified then the flag does nothing, as I assume is the case for
> 99% of people.
>

Indeed, that's been working fine for me for a while. However it's this
behavior that is actually the bug -- extensions are supposed to be build
against the 10.6 SDK -- and that's not what's happening in this case


> I understand the need to make Python 3.5 backwards compatible, but doing
> it this way would appear to have the side effect of breaking the ability to
> install packages using C extensions in virtual environments, or at least it
> did for me.
>

It does sound like you've hit a bug, but I actually think you've
misidentified the bug -- I'm trying to understand why it's not working in a
virtualenv -- THAT's the bug.

And I have no idea what to do about it -- thanks Apple for moving things --
that's a great way to keep backward compatibility!

Ned: I'm trying to explore this a bit now (on a 10.9 system), so I followed
the instructions in the Python developers guide, and did:

xcode-select --install

And was a bit surprised that It did a substantial download -- I thought I
already had the XCode command line tools -- but whatever. After running
that, I found I had the 10.9 and 10.10 SDKs (I think that installed the
10.10 one, I previously had only 10.9). And they are buried deep in
XCode.app 

OK, but how would I go about getting the 10.6 SDK??? This SO post:

http://stackoverflow.com/questions/11989007/where-to-get-macos-sdk-10-6-for-xcode

indicates that it's a serious pain in the $%^% -- thank Apple! But, as far
as I can tell, if you install an old XCode, you will get the older SDKs,
and installed into the old location.

(this is why I kept a 10.6 system around for a long time -- to build python
extensions...)

But what should we do? IF you are building extensions only for yourself,
then I think whatever the heck SDK you have will work fine -- but how to
build for distribution?

 Trying to decide if I want to go through the hassle of installing an
old XCode to see if i can replicate the bug the OP is reporting

-CHB


> https://docs.python.org/devguide/setup.html#build-dependencies
>




> >
> > Installing Xcode by itself is not sufficient.  The sysroot and flag
> values you note in _sysconfigdata.py are normal and there for compatibility
> with installing on older versions of OS X; they should not cause a problem
> on newer releases.  For what it's worth, I was able to install ZODB with
> the current python.org 3.5.1 on OS X 10.11.4 using either venv or
> virtualenv.
> >
> > Good luck!
> >
> > --Ned
> >
> > P.S. Mailing list info is available here: https://mail.python.org.  The
> bug tracker is here: https://bugs.python.org
> >
> > --
> >  Ned Deily
> >  n...@python.org -- []
> >
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 5

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Joseph Lee Nunn III]

Then I have found the right list to post on, for my problem is a bug in the 
Python 3.5.1 distribution.  Albeit one which will only manifest in certain 
situations.

The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` directives 
in the following file

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py

Will cause errors for people who have kept around the old /Developer directory 
tree, and old SDKs like 10.6, for building backwards compatible old code.  
Newer versions of the SDKs are now kept inside the Xcode application itself.  
And then they will only have a problem when trying to install a package with C 
extensions, which in my case was Persistence, a dependency of ZODB.  And even 
more so, ONLY when trying to pip install said package into a virtual 
environment created by 3.5’s pyvenv command, if installing the package into the 
system wide Python there will be no error.

Such is this corner-case hell I have found myself, until I decided to dump my 
older /Developer directory tree and SDKs, as those were for a project long 
defunct.  You see the -isysroot flag has the unusual behavior where instead of 
reporting an error when it cannot find the newly specified system root it 
simply has no effect.  So if you don’t have the SDK specified in the location 
specified then the flag does nothing, as I assume is the case for 99% of 
people. 

I understand the need to make Python 3.5 backwards compatible, but doing it 
this way would appear to have the side effect of breaking the ability to 
install packages using C extensions in virtual environments, or at least it did 
for me.

Joseph Nunn
josephn...@gmail.com
joseph.n...@uci.edu


> On May 06 , 2016, at 4:20 AM, Ned Deily  wrote:
> 
> 
> On May 6, 2016, at 05:42, Joseph Nunn  wrote:
>> Hello Everyone, my first post.  I would also like to subscribe but 
>> haven't seen how yet.  
>> 
>> So I've an issue with how the 3.5.1 OSX 
>> package installs on OSX.  The problem was first made apparent 
>> when trying to create a virtual environment and `pip install ZODB`, 
>> which failed as clang and the linker could not find either the 
>> include or lib directories for a dependency.  By adding a -v option
>> to the compiler options I was able to see in the  compiler flags 
>> the following:
>> 
>> -isysroot /Developer/SDKs/MacOSX10.6.sdk
>> 
>> Which was not present in the compiler flags of other people 
>> who I was talking with. [...]
> 
> Hi Joseph!
> 
> Welcome!  Note this list is for the development of Python itself, not for 
> usage or installation issues.  The Python Developer's Guide 
> (https://docs.python.org/devguide/) including its FAQ gives information about 
> mailing lists and other support forums to get help with general Python 
> issues.  That said, without more information, it would be difficult to know 
> exactly what problem you are running into but, since you mention missing 
> headers, my guess is that you need to install Apple's Command Line Tools as 
> described here:
> 
> https://docs.python.org/devguide/setup.html#build-dependencies
> 
> Installing Xcode by itself is not sufficient.  The sysroot and flag values 
> you note in _sysconfigdata.py are normal and there for compatibility with 
> installing on older versions of OS X; they should not cause a problem on 
> newer releases.  For what it's worth, I was able to install ZODB with the 
> current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.
> 
> Good luck!
> 
> --Ned
> 
> P.S. Mailing list info is available here: https://mail.python.org.  The bug 
> tracker is here: https://bugs.python.org
> 
> --
>  Ned Deily
>  n...@python.org -- []
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Ned Deily]

On May 6, 2016, at 05:42, Joseph Nunn  wrote:
> Hello Everyone, my first post.  I would also like to subscribe but 
> haven't seen how yet.  
> 
> So I've an issue with how the 3.5.1 OSX 
> package installs on OSX.  The problem was first made apparent 
> when trying to create a virtual environment and `pip install ZODB`, 
> which failed as clang and the linker could not find either the 
> include or lib directories for a dependency.  By adding a -v option
> to the compiler options I was able to see in the  compiler flags 
> the following:
> 
> -isysroot /Developer/SDKs/MacOSX10.6.sdk
> 
> Which was not present in the compiler flags of other people 
> who I was talking with. [...]

Hi Joseph!

Welcome!  Note this list is for the development of Python itself, not for usage 
or installation issues.  The Python Developer's Guide 
(https://docs.python.org/devguide/) including its FAQ gives information about 
mailing lists and other support forums to get help with general Python issues.  
That said, without more information, it would be difficult to know exactly what 
problem you are running into but, since you mention missing headers, my guess 
is that you need to install Apple's Command Line Tools as described here:

https://docs.python.org/devguide/setup.html#build-dependencies

Installing Xcode by itself is not sufficient.  The sysroot and flag values you 
note in _sysconfigdata.py are normal and there for compatibility with 
installing on older versions of OS X; they should not cause a problem on newer 
releases.  For what it's worth, I was able to install ZODB with the current 
python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.

Good luck!

--Ned

P.S. Mailing list info is available here: https://mail.python.org.  The bug 
tracker is here: https://bugs.python.org

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Wrong OSX platform in sysconfig.py causing installation problems

[Joseph Nunn]

Hello Everyone, my first post.  I would also like to subscribe but 
haven't seen how yet.  

So I've an issue with how the 3.5.1 OSX 
package installs on OSX.  The problem was first made apparent 
when trying to create a virtual environment and `pip install ZODB`, 
which failed as clang and the linker could not find either the 
include or lib directories for a dependency.  By adding a -v option
to the compiler options I was able to see in the  compiler flags 
the following:

-isysroot /Developer/SDKs/MacOSX10.6.sdk

Which was not present in the compiler flags of other people 
who I was talking with.  The isysroot flag changes the location 
of includes and libs from the system default, which would be 
why they can't be found.  After grepping around a bit I found 
this file:

/Library/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/_sysconfigdata.py

Within which are many mentions of this flag, as so:

'CONFIGURE_CFLAGS': '-arch i386 -arch x86_64 -isysroot 
/Developer/SDKs/MacOSX10.6.sdk -g',
'CONFIGURE_CFLAGS_NODIST': '-Werror=declaration-after
-statement',
'CONFIGURE_CPPFLAGS': 
'-isysroot /Developer/SDKs/MacOSX10.6.sdk',
'CONFIGURE_LDFLAGS': '-arch i386 -arch x86_64 -isysroot 
/Developer/SDKs/MacOSX10.6.sdk -g',
'CONFIG_ARGS': "'-C' '--enable-framework' '--enable-
universalsdk=/Developer/SDKs/MacOSX10.6.sdk' 
'--with-universal-archs=intel' 
'--with-computed-gotos' '--without-ensurepip' 'LDFLAGS=-g'
'CFLAGS=-g' 

'CC=gcc-4.2'",

among others.

By running Python 3.5's sysconfig.py as a module, via 
`python -m sysconfig`, I see the following at the beginning 
of the output:

Platform: "macosx-10.6-intel"
Python version: "3.5"
Current installation scheme: "posix_prefix"

And when I simply go into python interactive mode, import 
sysconfig, and type 

sysconfig.get_platform() it returns macosx-10.6-intel.

I believe somehow the platform being reported is incorrect, 
but looking at my environment, I do not have any kind of 
compiler flags or anything else like that set 

in my bash init files.

Can someone who has a working OSX 3.5 installation check 
their reported platform to see what it is and if anyone has an 
idea as to how this information is getting into the installation 
and where I might change it please let me know. 

Sincerely,

Joseph
josephn...@gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com