Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Paul Moore]

On 1 September 2015 at 17:15, Steve Dower  wrote:
> On 01Sep2015 0747, Oscar Benjamin wrote:
>>
>> Thanks for the detailed writeup Steve. Do you know how these changes
>> to the python.org Windows binaries would impact on people building
>> extension modules with MinGW?
>
>
> Currently, no version of MinGW AFAIK will link against the UCRT, so they'll
> suffer from the same mixed-CRT issues as with any other arrangement. There
> is some work going towards making mingw-w64 work with UCRT, but I am not
> following it closely despite occasional contact with the dev(s) working on
> it.

One thing that I hit when trying to build vim with VS2015 is that it
appears that even compiled object code isn't linkable with the ucrt. I
don't know the correct terminology here, so let me explain in a bit
more detail (I'm doing this from memory, so some symbol names might be
wrong, but you'll get the idea, I hope).

If source code refers to FILE* objects, then previously the compiled
object file (compiled with mingw) would be linkable with any of the
various C runtimes (msvcrt, msvcr10, etc). The object referred to a
symbol __iob_base which is present in all the CRTs. If the actual code
using the library doesn't actually use any of the functions that refer
to stdio, then whether the actual use of that symbol is broken doesn't
(seem to) matter in practice.

That's likely to be luck, to a certain extent - there's no guarantee
that the internals of the FILE* abstraction are compatible between
CRTs, but it has been true thus far, and so use of object libraries
built with mingw are typically OK, as long as you avoid using things
that need the CRT types like FILE*.

But with VS 2015 and the ucrt, there is no __iob_base symbol, and so
linking to code that refers to stdio fails.

The consequence of this (I believe) is that not only does mingw need
to be able to link against the ucrt, but it also needs to *compile*
with ucrt-compatible headers. In particular, when compiling for the
ucrt, mingw needs to use a version of  that doesn't reference
__iob_base. As the mingw developers maintain their own headers by some
process of "clean room" reimplementation (to satisfy the needs of the
GPL, as I understand it) updating the headers to cater for the ucrt
could be a non-trivial issue for them. I have no idea what their plans
are in this regard.

The other side of this is that it means that all object libraries you
use need to be recompiled to target ucrt, you can't simply use
existing compiled libraries and relink.

I hope this makes sense, and my interpretation is accurate - if I've
misunderstood the implications of the switch to the ucrt, then please
let me know. And if I'm wrong about not being able to use existing
compiled libraries, I would definitely appreciate someone telling me
what I'm doing wrong, because at the moment I'm unable to build my own
copy of vim with VS 2015 because I can't rebuild the xpm library :-(

Paul

PS My ultimate goal with building Vim with VS 2015 is to create a
build that uses the new embeddable distribution of Python 3.5 to
create a self-contained copy of Vim with Python 3.5 support enabled.
Something I think would be awesome :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Paul Moore]

On 2 September 2015 at 13:01, Carl Kleffner  wrote:
> that is https://bugs.python.org/msg248716 see also
> http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest

Interesting. But it does confirm that object files have to be rebuilt,
and there's no way to use existing object files. So if your source
only builds with mingw, it needs mingw's headers updating before you
can do that rebuild.

FWIW, in the case I hit with xpm, the xpm library only uses FILE*
internally - the external APIs are purely in terms of filenames. And
it only uses FILE* via the standard C APIs for it. So it's not a
simple case of "avoid using CRT types in external APIs".

So unless I'm misunderstanding, *any* object code used in a project
(whether directly or from a 3rd party library) has to be build using a
compiler that supports the ucrt (which, given that mingw doesn't have
that support yet, means VS 2015 only).

Note: I understand why the move to the ucrt is good, and I view this
as very much a short term transitional issue, but it is real and I
think it's important people understand the implications.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Carl Kleffner]

Hi,

that is https://bugs.python.org/msg248716 see also
http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest

Steve: is there more of that in the new universal runtimes?

Carl

2015-09-02 12:16 GMT+02:00 Paul Moore :

> On 1 September 2015 at 17:15, Steve Dower  wrote:
> > On 01Sep2015 0747, Oscar Benjamin wrote:
> >>
> >> Thanks for the detailed writeup Steve. Do you know how these changes
> >> to the python.org Windows binaries would impact on people building
> >> extension modules with MinGW?
> >
> >
> > Currently, no version of MinGW AFAIK will link against the UCRT, so
> they'll
> > suffer from the same mixed-CRT issues as with any other arrangement.
> There
> > is some work going towards making mingw-w64 work with UCRT, but I am not
> > following it closely despite occasional contact with the dev(s) working
> on
> > it.
>
> One thing that I hit when trying to build vim with VS2015 is that it
> appears that even compiled object code isn't linkable with the ucrt. I
> don't know the correct terminology here, so let me explain in a bit
> more detail (I'm doing this from memory, so some symbol names might be
> wrong, but you'll get the idea, I hope).
>
> If source code refers to FILE* objects, then previously the compiled
> object file (compiled with mingw) would be linkable with any of the
> various C runtimes (msvcrt, msvcr10, etc). The object referred to a
> symbol __iob_base which is present in all the CRTs. If the actual code
> using the library doesn't actually use any of the functions that refer
> to stdio, then whether the actual use of that symbol is broken doesn't
> (seem to) matter in practice.
>
> That's likely to be luck, to a certain extent - there's no guarantee
> that the internals of the FILE* abstraction are compatible between
> CRTs, but it has been true thus far, and so use of object libraries
> built with mingw are typically OK, as long as you avoid using things
> that need the CRT types like FILE*.
>
> But with VS 2015 and the ucrt, there is no __iob_base symbol, and so
> linking to code that refers to stdio fails.
>
> The consequence of this (I believe) is that not only does mingw need
> to be able to link against the ucrt, but it also needs to *compile*
> with ucrt-compatible headers. In particular, when compiling for the
> ucrt, mingw needs to use a version of  that doesn't reference
> __iob_base. As the mingw developers maintain their own headers by some
> process of "clean room" reimplementation (to satisfy the needs of the
> GPL, as I understand it) updating the headers to cater for the ucrt
> could be a non-trivial issue for them. I have no idea what their plans
> are in this regard.
>
> The other side of this is that it means that all object libraries you
> use need to be recompiled to target ucrt, you can't simply use
> existing compiled libraries and relink.
>
> I hope this makes sense, and my interpretation is accurate - if I've
> misunderstood the implications of the switch to the ucrt, then please
> let me know. And if I'm wrong about not being able to use existing
> compiled libraries, I would definitely appreciate someone telling me
> what I'm doing wrong, because at the moment I'm unable to build my own
> copy of vim with VS 2015 because I can't rebuild the xpm library :-(
>
> Paul
>
> PS My ultimate goal with building Vim with VS 2015 is to create a
> build that uses the new embeddable distribution of Python 3.5 to
> create a self-contained copy of Vim with Python 3.5 support enabled.
> Something I think would be awesome :-)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Carl Kleffner]

There are more semantic surprises as well. Some Math functions like powf,
acosf and so on are available only with the help of the VS 2015 math.h
include file (32 bit case). With exporting symbols names from the universal
dll's and creating import libraries you are not done yet.

Carl

2015-09-02 14:38 GMT+02:00 Paul Moore :

> On 2 September 2015 at 13:01, Carl Kleffner  wrote:
> > that is https://bugs.python.org/msg248716 see also
> > http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest
>
> Interesting. But it does confirm that object files have to be rebuilt,
> and there's no way to use existing object files. So if your source
> only builds with mingw, it needs mingw's headers updating before you
> can do that rebuild.
>
> FWIW, in the case I hit with xpm, the xpm library only uses FILE*
> internally - the external APIs are purely in terms of filenames. And
> it only uses FILE* via the standard C APIs for it. So it's not a
> simple case of "avoid using CRT types in external APIs".
>
> So unless I'm misunderstanding, *any* object code used in a project
> (whether directly or from a 3rd party library) has to be build using a
> compiler that supports the ucrt (which, given that mingw doesn't have
> that support yet, means VS 2015 only).
>
> Note: I understand why the move to the ucrt is good, and I view this
> as very much a short term transitional issue, but it is real and I
> think it's important people understand the implications.
>
> Paul
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Steve Dower]

You can also build existing object or static libraries into their own DLL with 
the old compiler and dynamically link to them. It's not perfect, but it's no 
worse than trying to link them in directly.

Otherwise, your analysis is correct. The OpenSSL issue was because they 
directly referred to __iob_func in their own code. It submitted a patch for it 
months ago and they fixed it, but possibly only in 1.0.2.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" 
Sent: ‎9/‎2/‎2015 5:38
To: "Carl Kleffner" 
Cc: "Steve Dower" ; "Python Dev" 
Subject: Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

On 2 September 2015 at 13:01, Carl Kleffner  wrote:
> that is https://bugs.python.org/msg248716 see also
> http://rt.openssl.org/Ticket/Display.html?id=3390&user=guest&pass=guest

Interesting. But it does confirm that object files have to be rebuilt,
and there's no way to use existing object files. So if your source
only builds with mingw, it needs mingw's headers updating before you
can do that rebuild.

FWIW, in the case I hit with xpm, the xpm library only uses FILE*
internally - the external APIs are purely in terms of filenames. And
it only uses FILE* via the standard C APIs for it. So it's not a
simple case of "avoid using CRT types in external APIs".

So unless I'm misunderstanding, *any* object code used in a project
(whether directly or from a 3rd party library) has to be build using a
compiler that supports the ucrt (which, given that mingw doesn't have
that support yet, means VS 2015 only).

Note: I understand why the move to the ucrt is good, and I view this
as very much a short term transitional issue, but it is real and I
think it's important people understand the implications.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Paul Moore]

On 2 September 2015 at 14:07, Steve Dower  wrote:
> You can also build existing object or static libraries into their own DLL
> with the old compiler and dynamically link to them. It's not perfect, but
> it's no worse than trying to link them in directly.

Interesting approach. "gcc -shared -o xpm.dll xpm.a" does about what I
want. Some thoughts:

1. This dynamically links to msvcrt.dll. Is that OK? I guess the
answer is "it's no worse than using such a DLL with Python 3.4 would
be" :-)
2. I presumably need an import lib. I can get gcc to generate a .a
format one, I'll need to see if VC 2015 can handle those, or if
there's a way to convert them (I'm sure there is, but it's been a long
time since I had to do that...)

Anyway, thanks for the response.

Just one further question if I may - on going the other way. Is it
acceptable to embed Python 3.5 (via fully runtime
LoadLibrary/GetProcAddress calls to load python35.dll) in an
application that uses an older CRT? My initial instinct is that it
probably isn't, but I can't think it through - my head's definitely
hurting by now :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building Extensions for Python 3.5 on Windows

[Steve Dower]

On 02Sep2015 0803, Paul Moore wrote:

On 2 September 2015 at 14:07, Steve Dower  wrote:

You can also build existing object or static libraries into their own DLL
with the old compiler and dynamically link to them. It's not perfect, but
it's no worse than trying to link them in directly.


Interesting approach. "gcc -shared -o xpm.dll xpm.a" does about what I
want. Some thoughts:

1. This dynamically links to msvcrt.dll. Is that OK? I guess the
answer is "it's no worse than using such a DLL with Python 3.4 would
be" :-)


Yeah, "no worse" is the answer here.


2. I presumably need an import lib. I can get gcc to generate a .a
format one, I'll need to see if VC 2015 can handle those, or if
there's a way to convert them (I'm sure there is, but it's been a long
time since I had to do that...)


I'm fairly sure there's a dumpbin.exe/lib.exe dance you can do, but it's 
been a long time since I've had to do it too.



Anyway, thanks for the response.

Just one further question if I may - on going the other way. Is it
acceptable to embed Python 3.5 (via fully runtime
LoadLibrary/GetProcAddress calls to load python35.dll) in an
application that uses an older CRT? My initial instinct is that it
probably isn't, but I can't think it through - my head's definitely
hurting by now :-)


The answer is "yes, but", where the but depends on what you're sharing 
between the application and Python. As long as you're careful to let 
Python do its memory management and the app do its own and their paths 
never cross, you'll probably be okay.


Cheers,
Steve


Paul



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] python programmer

[Linda Ryan]

Dear Admin,

I am an IT/Project Management recruiter looking to increase the available pool 
of talent for available job placements.
Currently I have an opening for a python programmer/developer.  Could I post 
opportunities to your member list?

Thank you,

Linda Ryan
Business Development Manager
770-313-2739 cell

TenStep Inc
www.TenStep.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python programmer

[Mark Shannon]

In a word, No.
Try https://www.python.org/jobs/

On 02/09/15 21:57, Linda Ryan wrote:

Dear Admin,

I am an IT/Project Management recruiter looking to increase the
available pool of talent for available job placements.
Currently I have an opening for a python programmer/developer.  Could I
post opportunities to your member list?

Thank you,

Linda Ryan
Business Development Manager
770-313-2739 cell

TenStep Inc
www.TenStep.com 




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python programmer

[Ethan Furman]

On 09/02/2015 02:53 PM, Mark Shannon wrote:

On 02/09/15 21:57, Linda Ryan wrote:


Could I post opportunities to your member list?


In a word, No.
Try https://www.python.org/jobs/


But thank you for asking first.  :)  Good luck!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python programmer

[Chris Angelico]

On Thu, Sep 3, 2015 at 6:57 AM, Linda Ryan  wrote:
> I am an IT/Project Management recruiter looking to increase the available
> pool of talent for available job placements.
> Currently I have an opening for a python programmer/developer.  Could I post
> opportunities to your member list?

Hi! Not to this list, as such, but there is a job board that accepts
such postings:

https://www.python.org/jobs/

People around the world read that board (myself included), so if
you're willing to work with telecommuters, your pool of talent can be
pretty broad!

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Yury Selivanov]

On 2015-07-06 11:38 AM, David Mertz wrote:

Hi Folks,

I hereby volunteer to write "What's New for Python 3.5?" if folks on 
python-dev are fine with me taking the job (i.e. I ran it by Travis, 
my boss at Continuum, and he's happy to allow me to do that work 
within my salaried hours... so having time isn't a problem).


If this is OK with the powers-that-be, I'll coordinate with David 
Murray on how best to take over this task from him.


Hi David,

Are you still going to work on what's new for 3.5?

Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python programmer

[Stefan Behnel]

Chris Angelico schrieb am 02.09.2015 um 23:54:
> On Thu, Sep 3, 2015 at 6:57 AM, Linda Ryan wrote:
>> I am an IT/Project Management recruiter looking to increase the available
>> pool of talent for available job placements.
>> Currently I have an opening for a python programmer/developer.  Could I post
>> opportunities to your member list?
> 
> Hi! Not to this list, as such, but there is a job board that accepts
> such postings:
> 
> https://www.python.org/jobs/
> 
> People around the world read that board (myself included), so if
> you're willing to work with telecommuters, your pool of talent can be
> pretty broad!

And if not, then your target audience might still read it there. However,
depending on where your position is, you might also be able to find a more
local or at least national job board, or a local Python User Group that
might be able to tell you where best to publish job offers for your target
group. Searching for "Python User Group" and your (closest bigger) city
name might reveal something there.

Good luck!

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com