Re: [Python-Dev] VS 2010 compiler

[Matthew Einhorn]

On Wed, Sep 30, 2015 at 3:57 PM, Carl Kleffner  wrote:

> Concerning the claims that mingw is difficult:
>
> The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain,
> that is in development. It is designed for simple use and for much better
> compatibility to the standard MSVC python builds. It should work out of the
> box, as long as the \Scripts folder is in the PATH.
>

Indeed, I tested it with 2.7.9 x86/x64, and wheres before I had to apply at
least http://bugs.python.org/issue4709, and
http://bugs.python.org/issue16472 and then generate the .a files to get it
to work, this worked out of the box! (except of course still needing to
change distutils.cfg).

I do wonder though, which may be slightly off topic here, whether putting
the mingw binary stub files (?) in python/Scripts, rather than directly
having python/shared/mingwpy/bin be on the path, is the best approach? I
suppose it's no worse than having the other stuff in python/scripts on the
path (although with e.g. pip you could do the python -m pip trick). Are
there guidelines about this (using python/scripts) for Windows? I didn't
see anything mentioned when python -m pip install ... vs. pip install is
discussed.

Although I hope a 7z version of that mingw will be separately available for
those that want to share a copy between versions of python, since the files
in share/mingwpy is presumably the same between python versions for a
specific bitness?

Matt
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker]

On Wed, Sep 30, 2015 at 12:15 PM, Paul Moore  wrote:

> > This, unfortunately is non-trivial, and really a pain if you want to
> > automate builds.
>
> Please clarify.


First point -- that was intended to be a lament, not a criticism. And
certainly not a criticism of anything Python devs are doing / have done.
It's more difficult than it "should" be because of how MS distributes and
configures the compiler and SDK, and that they are no longer distributing
the same thing they used to distribute (i.e. VS2010 Express).

I'm not at all sure that there is anything we can do to help

But once upon a time, it really was as simple as:

1) Install this freely available compiler from MS
2)  $ python setup.py build

and that is the case now with py2.7 and (I assume) py3.5

Compared to that, setting environment variables, having to use a particular
command prompt, etc. is non trivial.

Part of the problem is that there are multiple audiences here:

1) experienced Windows devs that are writing and building their own
packages:
  - not a big deal there, though it's still hard to figure out what is the
"right" way to do it -- adding this to the packaging docs will help, I hope.

2) developers on other platforms that want to be able to build and
distribute their package for Windows users.
  - this is a semi-big deal for them -- they need to learn a bit about how
Windows works, and will find that annoying and frustrating because "why
can't this be as easy as Linux?" (or really "the same as Linux") ;-)

3) users that want to be able to "pip install", and find that there is no
binary wheel available, and they get the dreaded "can't find vcvarsall.bat"
message.
  - these are the hardest people to support -- they may know very, very
little about Windows command lines and environment variables etc.

4) users that are not particularly familiar with command lines or
development tools, but need a little C extension -- Cython, in particular,
makes it really easy to do that -- but then you need to buld the darn
thing. distutils (setuptools) makes it possible to do that with very, very
little understanding of build tools, but you have to get set up first...


The whole process (including finding these instructions) is non-trivial for
everyone of these use cases other than (1)[1].

What is non-trivial? Installing the SDK? I know, but
> we said that's out of scope. Using an SDK command prompt? It is, sort
> of, particularly if (like me) you use powershell.


or gitBash :-)


> But again, not our
> issue. I assume setting the environment variable isn't an issue - you
> can do it for the session rather than globally, so even restrictive
> permissions aren't a problem.
>

but both of these add steps to the "$python setup.py install" -- that
really des make a difference to many users (see above)

I but it
> is an acknowledgement that often the audience for this sort of
> instruction are stumped by Microsoft's less than intuitive install
> processes...
>

yup !


> For context, installing mingw is just as messy, complicated and error
> prone (I speak from experience :-))


God yes!


> so it's unfair to complain that
> the above is a non-trivial pain. I know of no install option that's
> *less* straightforward than this (except of course for "install any
> version of Visual Studio 2010, even the free ones" - if you have
> access to those, use them!)
>

exactly! but that is the standard I'm comparing against.

When it comes to usability, there really is no argument about whether
something is "easy to use" -- As we've discussed, this comes up over an
over again on StackOverflow, various mailing lists, assorted Blogs, --
there really is no debate about this being "easy".

What can be done about it is a different question. My argument is that docs
can help, and maybe once this is in the packaging docs, all those mailing
list questions can be answered with a link to that.


> For automation, why not use Appveyor? See
> https://packaging.python.org/en/latest/appveyor/


yup -- very cool, I've been meaning to get that set up for distributing
some stuff.


> Unless you meant
> setting up a local build machine.


Exactly -- you need that while developing, anyway. And at the moment, I
have a non-sophisticated user-base that needs to be working with a
frequently updated gitHub repo -- so needs to build themselves.

(OK, if I had Appveyor doing updated binary packages every push, they
wouldn't need to build themselves -- maybe someday)



> If you want a simple "install a
> Python build environment" process, you could look at
> https://github.com/pfmoore/pybuild -


nice! I'll checdk that out. But I"m confused -- right in there, you write:


"setting up a Windows build environment can be a significant challenge"

then you ask me what is "non trivial" :-)

I haven't used it in a while (as
> it's of no relevance to me, because I have VS2010) but it does work. I
> never publicised or distributed it, because I got too much 

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 1 October 2015 at 18:18, Chris Barker  wrote:
>> If you want a simple "install a
>> Python build environment" process, you could look at
>> https://github.com/pfmoore/pybuild -
>
> nice! I'll checdk that out. But I"m confused -- right in there, you write:
>
> "setting up a Windows build environment can be a significant challenge"
>
> then you ask me what is "non trivial" :-)

You caught me out :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker]

I don't know that anyone disagrees with my point, but, less than an hour
ago, this on the wxPython list:

"""
Microsoft no longer sells the compiler that's needed to build Python 3.4,
and the needed compiler for Python 3.5 is free.
"""

To be fair, if you are trying to build wxWidgets, you may well want the
whole IDE, but still...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 30 September 2015 at 20:15, Paul Moore  wrote:
> I'll push an addition to packaging.python.org, probably tomorrow.

https://github.com/pypa/python-packaging-user-guide/pull/175

Unless there's a discussion on the PR, I'll probably commit it in a day or so.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 30 September 2015 at 16:57, Chris Barker - NOAA Federal
 wrote:
>> 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
>> 2. Work from an SDK command prompt (with the environment variables
>> set, and the SDK on PATH).
>> 3. Set DISTUTILS_USE_SDK=1
>> 4. Done.
>
> This, unfortunately is non-trivial, and really a pain if you want to
> automate builds.

Please clarify. What is non-trivial? Installing the SDK? I know, but
we said that's out of scope. Using an SDK command prompt? It is, sort
of, particularly if (like me) you use powershell. But again, not our
issue. I assume setting the environment variable isn't an issue - you
can do it for the session rather than globally, so even restrictive
permissions aren't a problem.

I appreciate you mightn't be intending this as criticism of the
instructions, but many people do criticise in exactly this sort of
way. Unix developers, in particular, who have limited Windows
knowledge, find this level of instruction really frustrating to deal
with. That's not a complaint - I have *huge* appreciation for
non-Windows users who bother to make builds for Windows users - but it
is an acknowledgement that often the audience for this sort of
instruction are stumped by Microsoft's less than intuitive install
processes...

For context, installing mingw is just as messy, complicated and error
prone (I speak from experience :-)) so it's unfair to complain that
the above is a non-trivial pain. I know of no install option that's
*less* straightforward than this (except of course for "install any
version of Visual Studio 2010, even the free ones" - if you have
access to those, use them!)

For automation, why not use Appveyor? See
https://packaging.python.org/en/latest/appveyor/ Unless you meant
setting up a local build machine. If you want a simple "install a
Python build environment" process, you could look at
https://github.com/pfmoore/pybuild - I haven't used it in a while (as
it's of no relevance to me, because I have VS2010) but it does work. I
never publicised or distributed it, because I got too much pushback in
terms of "but it doesn't work right on my system" (typically because
the system in question usually *wasn't* a clean build of Windows) that
I didn't have time or energy to address. But if it works for you, go
for it.

I'll push an addition to packaging.python.org, probably tomorrow.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Carl Kleffner]

Concerning the claims that mingw is difficult:

The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain,
that is in development. It is designed for simple use and for much better
compatibility to the standard MSVC python builds. It should work out of the
box, as long as the \Scripts folder is in the PATH.

It is not 'officially' released and announced, due to the fact that some
features are missing, the documentation has to be written and the build
scripts for the toolchain are not (yet) published.

Install a prerelease of mingwpy with pip:

  pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy

or with conda: (thanks to omnia-md)

  conda install -c https://conda.anaconda.org/omnia mingwpy

and use it at usual with pip install or python setup.py

You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy if
you have an MSVC compiler installed:

[config]
compiler=mingw32
[build]
compiler=mingw32
[build_ext]
compiler=mingw32

Or you install the latest portable winpython distribution
https://winpython.github.io that contains the toolchain as well and works
out of the box.

Future releases of mingwpy will be deployed on Pypi.

That has to be said: the main emphasis of the toolchain is building python
binary extension (C, C++, GFORTRAN) on windows, not building python itself.

Carl

2015-09-30 21:15 GMT+02:00 Paul Moore :

> On 30 September 2015 at 16:57, Chris Barker - NOAA Federal
>  wrote:
> >> 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
> >> 2. Work from an SDK command prompt (with the environment variables
> >> set, and the SDK on PATH).
> >> 3. Set DISTUTILS_USE_SDK=1
> >> 4. Done.
> >
> > This, unfortunately is non-trivial, and really a pain if you want to
> > automate builds.
>
> Please clarify. What is non-trivial? Installing the SDK? I know, but
> we said that's out of scope. Using an SDK command prompt? It is, sort
> of, particularly if (like me) you use powershell. But again, not our
> issue. I assume setting the environment variable isn't an issue - you
> can do it for the session rather than globally, so even restrictive
> permissions aren't a problem.
>
> I appreciate you mightn't be intending this as criticism of the
> instructions, but many people do criticise in exactly this sort of
> way. Unix developers, in particular, who have limited Windows
> knowledge, find this level of instruction really frustrating to deal
> with. That's not a complaint - I have *huge* appreciation for
> non-Windows users who bother to make builds for Windows users - but it
> is an acknowledgement that often the audience for this sort of
> instruction are stumped by Microsoft's less than intuitive install
> processes...
>
> For context, installing mingw is just as messy, complicated and error
> prone (I speak from experience :-)) so it's unfair to complain that
> the above is a non-trivial pain. I know of no install option that's
> *less* straightforward than this (except of course for "install any
> version of Visual Studio 2010, even the free ones" - if you have
> access to those, use them!)
>
> For automation, why not use Appveyor? See
> https://packaging.python.org/en/latest/appveyor/ Unless you meant
> setting up a local build machine. If you want a simple "install a
> Python build environment" process, you could look at
> https://github.com/pfmoore/pybuild - I haven't used it in a while (as
> it's of no relevance to me, because I have VS2010) but it does work. I
> never publicised or distributed it, because I got too much pushback in
> terms of "but it doesn't work right on my system" (typically because
> the system in question usually *wasn't* a clean build of Windows) that
> I didn't have time or energy to address. But if it works for you, go
> for it.
>
> I'll push an addition to packaging.python.org, probably tomorrow.
>
> Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Laurent Gautier]

Hi Carl,

Looks promising.

Any chance the effort would consider cross-compiling (from Linux) as a
possible objective ?

Best,

Laurent

On Wed, Sep 30, 2015, 3:58 PM Carl Kleffner  wrote:

> Concerning the claims that mingw is difficult:
>
> The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain,
> that is in development. It is designed for simple use and for much better
> compatibility to the standard MSVC python builds. It should work out of the
> box, as long as the \Scripts folder is in the PATH.
>
> It is not 'officially' released and announced, due to the fact that some
> features are missing, the documentation has to be written and the build
> scripts for the toolchain are not (yet) published.
>
> Install a prerelease of mingwpy with pip:
>
>   pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
>
> or with conda: (thanks to omnia-md)
>
>   conda install -c https://conda.anaconda.org/omnia mingwpy
>
> and use it at usual with pip install or python setup.py
>
> You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy if
> you have an MSVC compiler installed:
>
> [config]
> compiler=mingw32
> [build]
> compiler=mingw32
> [build_ext]
> compiler=mingw32
>
> Or you install the latest portable winpython distribution
> https://winpython.github.io that contains the toolchain as well and works
> out of the box.
>
> Future releases of mingwpy will be deployed on Pypi.
>
> That has to be said: the main emphasis of the toolchain is building python
> binary extension (C, C++, GFORTRAN) on windows, not building python itself.
>
> Carl
>
> 2015-09-30 21:15 GMT+02:00 Paul Moore :
>
>> On 30 September 2015 at 16:57, Chris Barker - NOAA Federal
>>  wrote:
>> >> 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
>> >> 2. Work from an SDK command prompt (with the environment variables
>> >> set, and the SDK on PATH).
>> >> 3. Set DISTUTILS_USE_SDK=1
>> >> 4. Done.
>> >
>> > This, unfortunately is non-trivial, and really a pain if you want to
>> > automate builds.
>>
>> Please clarify. What is non-trivial? Installing the SDK? I know, but
>> we said that's out of scope. Using an SDK command prompt? It is, sort
>> of, particularly if (like me) you use powershell. But again, not our
>> issue. I assume setting the environment variable isn't an issue - you
>> can do it for the session rather than globally, so even restrictive
>> permissions aren't a problem.
>>
>> I appreciate you mightn't be intending this as criticism of the
>> instructions, but many people do criticise in exactly this sort of
>> way. Unix developers, in particular, who have limited Windows
>> knowledge, find this level of instruction really frustrating to deal
>> with. That's not a complaint - I have *huge* appreciation for
>> non-Windows users who bother to make builds for Windows users - but it
>> is an acknowledgement that often the audience for this sort of
>> instruction are stumped by Microsoft's less than intuitive install
>> processes...
>>
>> For context, installing mingw is just as messy, complicated and error
>> prone (I speak from experience :-)) so it's unfair to complain that
>> the above is a non-trivial pain. I know of no install option that's
>> *less* straightforward than this (except of course for "install any
>> version of Visual Studio 2010, even the free ones" - if you have
>> access to those, use them!)
>>
>> For automation, why not use Appveyor? See
>> https://packaging.python.org/en/latest/appveyor/ Unless you meant
>> setting up a local build machine. If you want a simple "install a
>> Python build environment" process, you could look at
>> https://github.com/pfmoore/pybuild - I haven't used it in a while (as
>> it's of no relevance to me, because I have VS2010) but it does work. I
>> never publicised or distributed it, because I got too much pushback in
>> terms of "but it doesn't work right on my system" (typically because
>> the system in question usually *wasn't* a clean build of Windows) that
>> I didn't have time or energy to address. But if it works for you, go
>> for it.
>>
>> I'll push an addition to packaging.python.org, probably tomorrow.
>>
>> Paul
>>
> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Carl Kleffner]

It should be possible with Wine.
Am 30.09.2015 22:50 schrieb "Laurent Gautier" :

> Hi Carl,
>
> Looks promising.
>
> Any chance the effort would consider cross-compiling (from Linux) as a
> possible objective ?
>
> Best,
>
> Laurent
>
> On Wed, Sep 30, 2015, 3:58 PM Carl Kleffner  wrote:
>
>> Concerning the claims that mingw is difficult:
>>
>> The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain,
>> that is in development. It is designed for simple use and for much better
>> compatibility to the standard MSVC python builds. It should work out of the
>> box, as long as the \Scripts folder is in the PATH.
>>
>> It is not 'officially' released and announced, due to the fact that some
>> features are missing, the documentation has to be written and the build
>> scripts for the toolchain are not (yet) published.
>>
>> Install a prerelease of mingwpy with pip:
>>
>>   pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
>>
>> or with conda: (thanks to omnia-md)
>>
>>   conda install -c https://conda.anaconda.org/omnia mingwpy
>>
>> and use it at usual with pip install or python setup.py
>>
>> You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy
>> if you have an MSVC compiler installed:
>>
>> [config]
>> compiler=mingw32
>> [build]
>> compiler=mingw32
>> [build_ext]
>> compiler=mingw32
>>
>> Or you install the latest portable winpython distribution
>> https://winpython.github.io that contains the toolchain as well and
>> works out of the box.
>>
>> Future releases of mingwpy will be deployed on Pypi.
>>
>> That has to be said: the main emphasis of the toolchain is building
>> python binary extension (C, C++, GFORTRAN) on windows, not building python
>> itself.
>>
>> Carl
>>
>> 2015-09-30 21:15 GMT+02:00 Paul Moore :
>>
>>> On 30 September 2015 at 16:57, Chris Barker - NOAA Federal
>>>  wrote:
>>> >> 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
>>> >> 2. Work from an SDK command prompt (with the environment variables
>>> >> set, and the SDK on PATH).
>>> >> 3. Set DISTUTILS_USE_SDK=1
>>> >> 4. Done.
>>> >
>>> > This, unfortunately is non-trivial, and really a pain if you want to
>>> > automate builds.
>>>
>>> Please clarify. What is non-trivial? Installing the SDK? I know, but
>>> we said that's out of scope. Using an SDK command prompt? It is, sort
>>> of, particularly if (like me) you use powershell. But again, not our
>>> issue. I assume setting the environment variable isn't an issue - you
>>> can do it for the session rather than globally, so even restrictive
>>> permissions aren't a problem.
>>>
>>> I appreciate you mightn't be intending this as criticism of the
>>> instructions, but many people do criticise in exactly this sort of
>>> way. Unix developers, in particular, who have limited Windows
>>> knowledge, find this level of instruction really frustrating to deal
>>> with. That's not a complaint - I have *huge* appreciation for
>>> non-Windows users who bother to make builds for Windows users - but it
>>> is an acknowledgement that often the audience for this sort of
>>> instruction are stumped by Microsoft's less than intuitive install
>>> processes...
>>>
>>> For context, installing mingw is just as messy, complicated and error
>>> prone (I speak from experience :-)) so it's unfair to complain that
>>> the above is a non-trivial pain. I know of no install option that's
>>> *less* straightforward than this (except of course for "install any
>>> version of Visual Studio 2010, even the free ones" - if you have
>>> access to those, use them!)
>>>
>>> For automation, why not use Appveyor? See
>>> https://packaging.python.org/en/latest/appveyor/ Unless you meant
>>> setting up a local build machine. If you want a simple "install a
>>> Python build environment" process, you could look at
>>> https://github.com/pfmoore/pybuild - I haven't used it in a while (as
>>> it's of no relevance to me, because I have VS2010) but it does work. I
>>> never publicised or distributed it, because I got too much pushback in
>>> terms of "but it doesn't work right on my system" (typically because
>>> the system in question usually *wasn't* a clean build of Windows) that
>>> I didn't have time or energy to address. But if it works for you, go
>>> for it.
>>>
>>> I'll push an addition to packaging.python.org, probably tomorrow.
>>>
>>> Paul
>>>
>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>>
>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
>>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
>>
>

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 30 September 2015 at 01:31, Chris Barker - NOAA Federal
 wrote:
> It may be -- I've still have to test. Neither Windows nor py3 are my
> primary production versions.

Part of the problem here is that people put up postings saying "do
this" having checked that it works for them, and then others say "but
it doesn't work", without giving any information as to why or
providing any means to improve the instructions.

I'm not suggesting here that you're going to do this - but those
instructions are in essence the "official" (not well documented,
conceded) advice at present, and they have been tested by a number of
people for whom they work. They have also been reported to fail, but
(as noted above) never in a way that allows us to improve them :-(
They will remain the official advice, I suspect, until we get better
feedback.

> But the last time I tried the SDK approach was for py2.7 on Win64, and
> it was both a pain, and impossible to set up to "just work" in my
> highly secure environment.

Well, I'd need more explanation of the issues. Was "install the SDK" a
problem? Or was "set the environment variable DISTUTILS_USE_SDK"? Or
did the instructions to do those two things not work? (I appreciate
we're talking about 2 different things here, and I think Python 3 is a
much less painful case than 2.7 on win64 was prior to the Visual C for
Python 2.7 package).

I suspect (from your mention of a highly secure environment) that
installing the SDK was the main pain point. That's been the case for a
number of people, I believe. But I don't think advice on how to
install the Windows SDK in various environments is something we'd want
to maintain in the Python documentation...

This of course is another aspect of the problem - the official answer
is "use a supported version of Windows, ensure you have the necessary
permissions to install the needed software, and off you go". But in
real life people want to build on an old laptop they had lying around,
or on a VM they created on Linux, or on their Linux box using Wine, or
on the box they use for a load of other development that has more SDKs
than you want to imagine already installed, some of which got broken
in the past, ... Or they need to use mingw, because a key library
can't be compiled with Visual Studio. These are all real problems, and
the people fighting them are doing a huge amount of good work building
binary distributions for the community. They document what they are
doing to solve their issues and those documents show up on Google,
causing people who just want to build a simple extension to find them
and get hopelessly confused.

> So at best, it needs some serious documentation. And no, it shouldn't
> be just a blog post somewhere (unless maybe Steve Dowers's blog).

OK, without wishing to minimise what I fully agree is a serious
problem, at a very high level the instructions are:

For Python 2.7:

1. Install "Visual C++ Compiler Package for Python 2.7"
2. Use (a recent version of) setuptools in your setup.py (pip will do
this for you, in any case).
3. Done.

For Python 3.4

1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
2. Work from an SDK command prompt (with the environment variables
set, and the SDK on PATH).
3. Set DISTUTILS_USE_SDK=1
4. Done.

For Python 3.5

1. Install Visual Studio 2015 Community Edition (or any later version,
when these are released).
2. Done.

> Thus whole issue has been dogging python on windows for many years --
> it never has been easy to find out what to do.
>
> In fact, there are all sorts of Blog posts out there, with various
> levels of out of date, conflicting, or just plain wrong advice.

And this is why it's hard to fix the problem - people don't look at
the documentation, they don't try to find "official" advice, they
simply google and get a load of confusing advice. See above as to why
some of those documents are there, and are more complex than the
average user needs.

> A definitive version in the official docs would be a very good thing.

Well, I'd happily put the above instructions in the packaging user
guide (packaging.python.org). Would that be sufficient for you?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker - NOAA Federal]

TL;DR -- yes, documenting this in packaging.python.org would be great!

> Part of the problem here is that people put up postings saying "do
> this" having checked that it works for them, and then others say "but
> it doesn't work", without giving any information as to why or
> providing any means to improve the instructions.

Exactly. Which is why a Blog post is not the answer. If I find a blog
post, I have no idea how well vetted and tested it is.

If you look at the history of the Cython Wiki page about this, it
becomes very clear how much confusion there is. (Though I think it's
in pretty god shape now-- wikis have the advantage of being able to
evolve and getter better).

> I'm not suggesting here that you're going to do this -

Absolutely not -- I would not post anything anywhere without review --
this's kind of the point. If I find it in an official source, I know
it's been reviewed.

> us to improve them :-(
> They will remain the official advice, I suspect, until we get better
> feedback.

Which is my point -- how do I currently know what the official advise
is? (Yes, I know now -- thanks!) But posting to python-dev is exactly
the wrong way to find out!

>> But the last time I tried the SDK approach was for py2.7 on Win64, and
>> it was both a pain, and impossible to set up to "just work" in my
>> highly secure environment.
>
> Well, I'd need more explanation of the issues. Was "install the SDK" a
> problem? Or was "set the environment variable DISTUTILS_USE_SDK"? Or
> did the instructions to do those two things not work?

The issue was not having permissions to alter the registry the way we
wanted. But again, I'll need to do some proper testing with up to date
everything and see what happens.

>  I think Python 3 is a
> much less painful case than 2.7 on win64 was prior to the Visual C for
> Python 2.7 package).

Good to know.

>  But I don't think advice on how to
> install the Windows SDK in various environments is something we'd want
> to maintain in the Python documentation...

Agreed -- if you can't install it you can't install it. And if it
can't be installed in user-space, that's between MS and your
sysadmins...

> This of course is another aspect of the problem - the official answer
> is "use a supported version of Windows, ensure you have the necessary
> permissions to install the needed software, and off you go".

Well, I for one think that's fine. Supporting open source software on
a closed-source OS is going to have these pain points.

> OK, without wishing to minimise what I fully agree is a serious
> problem, at a very high level the instructions are:
>
> For Python 2.7:
>
> 1. Install "Visual C++ Compiler Package for Python 2.7"
> 2. Use (a recent version of) setuptools in your setup.py (pip will do
> this for you, in any case).
> 3. Done.

That new MS installer is really really great.

> For Python 3.4
>
> 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1)
> 2. Work from an SDK command prompt (with the environment variables
> set, and the SDK on PATH).
> 3. Set DISTUTILS_USE_SDK=1
> 4. Done.

This, unfortunately is non-trivial, and really a pain if you want to
automate builds.

Apparently, conda-build has got this automated under the hood. Maybe
that could be kludge do into setuptools?

>  - people don't look at
> the documentation, they don't try to find "official" advice, they
> simply google and get a load of confusing advice.

Well, yes, but when there is no official advice, looking for it doesn't help

I, for one, have looked for docs and official advise over and over
again for many years ( back when kept trying to remember if py2.6 was
built with VS2003 or VS2008, or ??

> Well, I'd happily put the above instructions in the packaging user
> guide (packaging.python.org). Would that be sufficient for you?

That would be great, thanks! An "official" source, but easier to
update that the regular docs.

Thanks,

-Chris
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker]

 And in mingw-w64, the only way to select a non-default CRT
>
what is a "default" CRT in this case??

> CRT issues are indeed tricky, because they only bite in certain
> circumstances -- so long as you never pass a FILE* or a fileno across the
> dll boundary, or call malloc in one module and free in the other, or ...
>
Which happens to be EXACTLY what I'm needing to do right now

then it works fine. Except when it doesn't :-).
>
yup -- a new developer on my team spent yesterday bangin his head agaist
this -- then asked "I'm compiling the lib with VS2013, the the python
extension with VS2008, could that be the problem? -- uh, yes?

The Python core developers made the decision a long time ago to use the MS
compilers for the "official" distributions -- as they are the "platform"
compilers. And as far as I know, everyone else has followed that path for
compatibility (Enthought Canopy, Anaconda, ActiveState) (except cygWin, but
that's not really the Windows platform, IFAIC). Given MS decision to both:
tie the compiler tightly to the run-time, AND make it hard to get old
versions of the compiler, that's turned out to be a pain. But I suspect
there have been real advantages over, for instance, building the whole
thing on MinGW.

But in any case, that's what we've got -- and it would be really nice  to
support teh community as best we can, as has been done fro Python2.7 with
updates to setuptools and the MS download of the old VS2008.

It sounds like we don't expect MS t help out in this case, but at least we
can better document what users need to do, and how to do it -- and NOT with
a "kludge together an open-source compiler and cross your fingers" approach.

OK -- I'm going to get off my soap box now -- time to actually suggest doc
patches

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Steve Dower]

On 29Sep2015 0820, Chris Barker wrote:

OK -- I'm going to get off my soap box now -- time to actually suggest
doc patches


Just bear in mind that you're suggesting patches for Python 3.3 and 3.4, 
which means that 3.4.4 is the only real chance to get them onto people's 
machines. http://docs.python.org/ already points to the 3.5 docs by 
default, which means only people who explicitly look for the earlier 
versions will see the changes.


You don't really have a huge audience, so I wouldn't invest large 
amounts of time in the official docs. A couple of well-publicised blog 
posts will reach more people.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Oscar Benjamin]

On Tue, 29 Sep 2015 17:20 Steve Dower  wrote:

On 29Sep2015 0820, Chris Barker wrote:
> OK -- I'm going to get off my soap box now -- time to actually suggest
> doc patches

Just bear in mind that you're suggesting patches for Python 3.3 and 3.4,
which means that 3.4.4 is the only real chance to get them onto people's
machines. http://docs.python.org/ already points to the 3.5 docs by
default, which means only people who explicitly look for the earlier
versions will see the changes.

You don't really have a huge audience, so I wouldn't invest large
amounts of time in the official docs. A couple of well-publicised blog
posts will reach more people.

Maybe but I think it would be good to have some documentation somewhere
(perhaps under the pypa banner) that can advise extension module authors
who want to provide Windows binaries for multiple CPython versions. It's
not obvious how to go about setting things up to make python.org compatible
binaries for say Python 2.7 and 3.2-3.5. Before 3.5 the simplest solution
was to use MinGW but that doesn't work for 3.5 (hence numpy are currently
uploading rcs without 3.5 binaries for Windows).

Just an explanation of what it takes in terms of MS compilers and which
free as in beer ones are available where would be helpful which could go in
a blog post as you say. IMO It would be more helpful though if it were a
continuously improved document that will be updated as new CPython releases
come out and the availability of MS compilers changes.

--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker - NOAA Federal]

> I'm not sure why INADA Naoki's answer above wasn't sufficient for you?

It may be -- I've still have to test. Neither Windows nor py3 are my
primary production versions.

But the last time I tried the SDK approach was for py2.7 on Win64, and
it was both a pain, and impossible to set up to "just work" in my
highly secure environment.

So at best, it needs some serious documentation. And no, it shouldn't
be just a blog post somewhere (unless maybe Steve Dowers's blog).

Thus whole issue has been dogging python on windows for many years --
it never has been easy to find out what to do.

In fact, there are all sorts of Blog posts out there, with various
levels of out of date, conflicting, or just plain wrong advice.

A definitive version in the official docs would be a very good thing.

BTW, I think this is a good discussion:

https://github.com/cython/cython/wiki/CythonExtensionsOnWindows

But not everyone is going to think to look at the Cython Wiki...

-Chris


> You need to have the SDK compilers on your PATH (by running the
> relevant environment setting command) and you need to set the
> environment variable DISTUTILS_USE_SDK=1 (this latter bit of
> information is hard to find documented, admittedly, but pretty well
> known).
>
> That basically gives you a Visual Studio 2010 equivalent environment.
> You'll still have all sorts of *other* problems building extensions on
> Windows, but unless I'm missing something important in what you were
> saying, getting a compiler isn't the issue.
>
> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker - NOAA Federal]

> On Sep 29, 2015, at 9:20 AM, Steve Dower  wrote:
>
>> On 29Sep2015 0820, Chris Barker wrote:
>> OK -- I'm going to get off my soap box now -- time to actually suggest
>> doc patches
>
> Just bear in mind that you're suggesting patches for Python 3.3 and 3.4, 
> which means that 3.4.4 is the only real chance to get them onto people's 
> machines.

I don't expect to get it onto anyone's machine -- does anyone download
docs anymore?

> http://docs.python.org/ already points to the 3.5 docs by default, which 
> means only people who explicitly look for the earlier versions will see the 
> changes.

Well, if your trying to build an extension for 3.4, hopefully you'll
think to look at 3.4 docs.

But good point -- the 3.5 docs should have a note saying that the
requirements are different than older releases.

> You don't really have a huge audience, so I wouldn't invest large amounts of 
> time in the official docs. A couple of well-publicised blog posts will reach 
> more people.

Not on my blog.

At this point, I think the Cython Wiki us the best place to point
people, but really, less than ideal.

-CHB


>
> Cheers,
> Steve
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Nathaniel Smith]

On Sep 29, 2015 8:22 AM, "Chris Barker"  wrote:
>
>>  And in mingw-w64, the only way to select a non-default CRT
>
> what is a "default" CRT in this case??

To answer the question: mingw-w64's
standard/default/only-officially-supported CRT is msvcrt.dll (the version
that has no version number in the name). Wikipedia tells me that this is
the runtime used by msvc 4.2 - 6.0 (the later of which was released in
1998). The reason for this decision is that for whatever historical reason,
this particular CRT dll is shipped with Windows by default, and thus using
it avoids certain license issues that can otherwise arise, esp. with GPL
software. (Fortunately not an issue for python extensions, but since it is
an issue for other programs they just made the decision to support a single
configuration that works for everybody. Except when it doesn't ;-).)

In practice mingw-w64 ships its own runtime as a layer on top of msvcrt.dll
-- it gets used as a OS services layer, but mingw-w64 users aren't just
getting dropped directly into a wonderland of 15+ year old code.

(In fact, mingw-w64 upstream would like to make their runtime clever enough
to bind the underlying "OS services layer" at load time rather than compile
time, so that mingw-w64 compiled dll's would be "runtime agile" and could
be safely loaded into any process. The main thing we care about is just
that this would give upstream a way to officially support targeting newer
CRTs, but it would have a number of other benefits too: one modern
toolchain that could build extension modules for all versions of python,
the ability when combined with Python's "stable abi" to target 3.2+ with a
single binary distribution, better interoperability between python and
other windows ecosystems that went the mingw-w64 route like R... but, this
all requires funding someone to do the work.)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 25 September 2015 at 16:35, INADA Naoki  wrote:
> You can use "Windows SDK for Windows 7 and .NET Framework 4".
>
> http://www.microsoft.com/en-us/download/details.aspx?id=8279
>
> On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal
>  wrote:
>>
>> As I understand it, the MS VS2010 compiler is required (or at least
>> best practice) for compiling Python extensions for the python.org
>> Windows builds of py 3.4 and ?[1]
>>
>> However, MS now makes it very hard (impossible?) to download VS2010
>> Express ( or Community, or whatever the free as in beer version is
>> called).
>>
>> I realize that this is not python-dev's responsibility, but if there
>> is any way to either document where it can be found, or put a bit of
>> pressure on MS to make it available, as they have for VS2008 and
>> py2.7, that would be great.
>>
>> Sorry to bug this list, I didn't know where else to reach out to.

On 29 September 2015 at 16:20, Chris Barker  wrote:
> It sounds like we don't expect MS t help out in this case, but at least we
> can better document what users need to do, and how to do it -- and NOT with
> a "kludge together an open-source compiler and cross your fingers" approach.

I'm not sure why INADA Naoki's answer above wasn't sufficient for you?
You need to have the SDK compilers on your PATH (by running the
relevant environment setting command) and you need to set the
environment variable DISTUTILS_USE_SDK=1 (this latter bit of
information is hard to find documented, admittedly, but pretty well
known).

That basically gives you a Visual Studio 2010 equivalent environment.
You'll still have all sorts of *other* problems building extensions on
Windows, but unless I'm missing something important in what you were
saying, getting a compiler isn't the issue.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Erik Bray]

On Fri, Sep 25, 2015 at 6:27 PM, Chris Barker - NOAA Federal
 wrote:
>
> You can use "Windows SDK for Windows 7 and .NET Framework 4".
>
> http://www.microsoft.com/en-us/download/details.aspx?id=8279
>
>
> Thanks. Last time I tried that route, it was for 64 bit py2.7. And it
> required some kludging of environment variables, and registry acces I don't
> have permission for.
>
> But it still may be the best bet. I'll give it a try when I have a chance.
>
> And this should be in the "official" docs...

For what it's worth, I've had good luck compiling *most* extension
modules on Windows using the gcc from MinGW-w64.

The big notable exception was that last time I tried compiling Numpy
with it I got a segfaulting Numpy.  But I never had a chance to
investigate why or if it's fixable.  My own extension modules work
fine on Windows when compiled in MinGW though.

Erik B.


> On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal
>  wrote:
>>
>> As I understand it, the MS VS2010 compiler is required (or at least
>> best practice) for compiling Python extensions for the python.org
>> Windows builds of py 3.4 and ?[1]
>>
>> However, MS now makes it very hard (impossible?) to download VS2010
>> Express ( or Community, or whatever the free as in beer version is
>> called).
>>
>> I realize that this is not python-dev's responsibility, but if there
>> is any way to either document where it can be found, or put a bit of
>> pressure on MS to make it available, as they have for VS2008 and
>> py2.7, that would be great.
>>
>> Sorry to bug this list, I didn't know where else to reach out to.
>>
>> -Chris
>>
>> [1] it's actually prefer hard to find out which compiler version is
>> used for which python version. And has been for years. Would a patch
>> to the docs, probably here:
>>
>> https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
>>
>> Be considered?
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>
>
>
>
> --
> INADA Naoki  
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/erik.m.bray%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Nathaniel Smith]

On Sep 28, 2015 1:39 PM, "Paul Moore"  wrote:
>
> On 28 September 2015 at 21:18, MRAB  wrote:
> > Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
> > both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
> > that wasn't due to a bug in the source code.
>
> Interesting. With Python 3.5, what CRT does the module use? With
> Python 3.4 and earlier, distutils arranged for mingw builds to link to
> the CRT that matched the CPython build,

This doesn't sound right to me. Maybe if you're talking about the original
mingw, but that's been abandoned upstream for years. Nowadays everyone uses
the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way
to select a non-default CRT is to hack up the toolchain's source tree and
cross your fingers. It's definitely not a supported configuration upstream.
(Source: we've been talking to mingw-w64 upstream and MS about how to
potentially make building free software less painful on Windows.)

> but I don't think that
> happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
> have a CRT that doesn't match the core. Obviously this isn't causing
> you issues, but I'm not sure if it could (CRT problems seem to be
> mostly about "might cause issues" type problems...)

CRT issues are indeed tricky, because they only bite in certain
circumstances -- so long as you never pass a FILE* or a fileno across the
dll boundary, or call malloc in one module and free in the other, or ...
then it works fine. Except when it doesn't :-).

Example of a subtle issue: msvc and mingw-w64 disagree about whether the
x87 fpu should be configured in extended precision mode or not. So simply
loading a mingw-w64-compiled dll can potentially cause math functions in
other dlls to start returning incorrect results. Or maybe not -- it depends
on the details of how MS implemented things, and we can't exactly check. Or
whether you get correct math result in mingw-w64 compiled code can depend
on which runtime was used to spawn the thread that's running the code (!!).
And even if things work today, it could change without warning in any
update, because no one is testing this configuration upstream.

It's totally plausible that the regex module happens to avoid all these
issues. But you can't assume that it's a general solution.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[MRAB]

On 2015-09-28 19:00, Erik Bray wrote:

On Fri, Sep 25, 2015 at 6:27 PM, Chris Barker - NOAA Federal
 wrote:


You can use "Windows SDK for Windows 7 and .NET Framework 4".

http://www.microsoft.com/en-us/download/details.aspx?id=8279


Thanks. Last time I tried that route, it was for 64 bit py2.7. And it
required some kludging of environment variables, and registry acces I don't
have permission for.

But it still may be the best bet. I'll give it a try when I have a chance.

And this should be in the "official" docs...


For what it's worth, I've had good luck compiling *most* extension
modules on Windows using the gcc from MinGW-w64.

The big notable exception was that last time I tried compiling Numpy
with it I got a segfaulting Numpy.  But I never had a chance to
investigate why or if it's fixable.  My own extension modules work
fine on Windows when compiled in MinGW though.

Erik B.


Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
that wasn't due to a bug in the source code.



On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal
 wrote:


As I understand it, the MS VS2010 compiler is required (or at least
best practice) for compiling Python extensions for the python.org
Windows builds of py 3.4 and ?[1]

However, MS now makes it very hard (impossible?) to download VS2010
Express ( or Community, or whatever the free as in beer version is
called).

I realize that this is not python-dev's responsibility, but if there
is any way to either document where it can be found, or put a bit of
pressure on MS to make it available, as they have for VS2008 and
py2.7, that would be great.

Sorry to bug this list, I didn't know where else to reach out to.

-Chris

[1] it's actually prefer hard to find out which compiler version is
used for which python version. And has been for years. Would a patch
to the docs, probably here:

https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows

Be considered?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com





--
INADA Naoki  


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/erik.m.bray%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/python%40mrabarnett.plus.com




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[MRAB]

On 2015-09-28 21:38, Paul Moore wrote:

On 28 September 2015 at 21:18, MRAB  wrote:
> Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
> both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
> that wasn't due to a bug in the source code.

Interesting. With Python 3.5, what CRT does the module use? With
Python 3.4 and earlier, distutils arranged for mingw builds to link to
the CRT that matched the CPython build, but I don't think that
happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
have a CRT that doesn't match the core. Obviously this isn't causing
you issues, but I'm not sure if it could (CRT problems seem to be
mostly about "might cause issues" type problems...)


I can't remember the details off-hand, but here's the issue:

http://bugs.python.org/issue24385

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 28 September 2015 at 21:18, MRAB  wrote:
> Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
> both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
> that wasn't due to a bug in the source code.

Interesting. With Python 3.5, what CRT does the module use? With
Python 3.4 and earlier, distutils arranged for mingw builds to link to
the CRT that matched the CPython build, but I don't think that
happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
have a CRT that doesn't match the core. Obviously this isn't causing
you issues, but I'm not sure if it could (CRT problems seem to be
mostly about "might cause issues" type problems...)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Paul Moore]

On 28 September 2015 at 22:18, Nathaniel Smith  wrote:
> On Sep 28, 2015 1:39 PM, "Paul Moore"  wrote:
>>
>> On 28 September 2015 at 21:18, MRAB  wrote:
>> > Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
>> > both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
>> > that wasn't due to a bug in the source code.
>>
>> Interesting. With Python 3.5, what CRT does the module use? With
>> Python 3.4 and earlier, distutils arranged for mingw builds to link to
>> the CRT that matched the CPython build,
>
> This doesn't sound right to me. Maybe if you're talking about the original
> mingw, but that's been abandoned upstream for years. Nowadays everyone uses
> the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way
> to select a non-default CRT is to hack up the toolchain's source tree and
> cross your fingers. It's definitely not a supported configuration upstream.
> (Source: we've been talking to mingw-w64 upstream and MS about how to
> potentially make building free software less painful on Windows.)

Interesting, it's been a long time since I've done work on the mingw
support, so I hadn't realised this had happened.

I guess that means that nowadays all mingw-compiled extensions link to
msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?

>> but I don't think that
>> happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
>> have a CRT that doesn't match the core. Obviously this isn't causing
>> you issues, but I'm not sure if it could (CRT problems seem to be
>> mostly about "might cause issues" type problems...)
>
> CRT issues are indeed tricky, because they only bite in certain
> circumstances -- so long as you never pass a FILE* or a fileno across the
> dll boundary, or call malloc in one module and free in the other, or ...
> then it works fine. Except when it doesn't :-).
>
> Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87
> fpu should be configured in extended precision mode or not. So simply
> loading a mingw-w64-compiled dll can potentially cause math functions in
> other dlls to start returning incorrect results. Or maybe not -- it depends
> on the details of how MS implemented things, and we can't exactly check. Or
> whether you get correct math result in mingw-w64 compiled code can depend on
> which runtime was used to spawn the thread that's running the code (!!). And
> even if things work today, it could change without warning in any update,
> because no one is testing this configuration upstream.
>
> It's totally plausible that the regex module happens to avoid all these
> issues. But you can't assume that it's a general solution.

Agreed. That was essentially what I was asking - has someone found a
solution to the issue of avoiding mixed CRTs when using mingw as a
compiler (or at least, found a better solution than using the original
mingw -lmsvcr100 approach combined with the distutils code that
supports this). Looks like the answer is that nobody has. But it's
certainly (still) true that for straightforward cases things seem
pretty much OK, especially if you avoid the "obvious" issues. (The
non-obvious ones sound terrifying, and I'm glad I don't have to write
code that encounters those!!!)

Cheers,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Mark Lawrence]

On 29/09/2015 00:52, Nathaniel Smith wrote:

On Mon, Sep 28, 2015 at 2:55 PM, Paul Moore  wrote:

On 28 September 2015 at 22:18, Nathaniel Smith  wrote:

On Sep 28, 2015 1:39 PM, "Paul Moore"  wrote:


On 28 September 2015 at 21:18, MRAB  wrote:

Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
that wasn't due to a bug in the source code.


Interesting. With Python 3.5, what CRT does the module use? With
Python 3.4 and earlier, distutils arranged for mingw builds to link to
the CRT that matched the CPython build,


This doesn't sound right to me. Maybe if you're talking about the original
mingw, but that's been abandoned upstream for years. Nowadays everyone uses
the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way
to select a non-default CRT is to hack up the toolchain's source tree and
cross your fingers. It's definitely not a supported configuration upstream.
(Source: we've been talking to mingw-w64 upstream and MS about how to
potentially make building free software less painful on Windows.)


Interesting, it's been a long time since I've done work on the mingw
support, so I hadn't realised this had happened.

I guess that means that nowadays all mingw-compiled extensions link to
msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?


Right, linking to msvcrt.dll is the only standard/supported
configuration for upstream mingw-w64.


but I don't think that
happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
have a CRT that doesn't match the core. Obviously this isn't causing
you issues, but I'm not sure if it could (CRT problems seem to be
mostly about "might cause issues" type problems...)


CRT issues are indeed tricky, because they only bite in certain
circumstances -- so long as you never pass a FILE* or a fileno across the
dll boundary, or call malloc in one module and free in the other, or ...
then it works fine. Except when it doesn't :-).

Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87
fpu should be configured in extended precision mode or not. So simply
loading a mingw-w64-compiled dll can potentially cause math functions in
other dlls to start returning incorrect results. Or maybe not -- it depends
on the details of how MS implemented things, and we can't exactly check. Or
whether you get correct math result in mingw-w64 compiled code can depend on
which runtime was used to spawn the thread that's running the code (!!). And
even if things work today, it could change without warning in any update,
because no one is testing this configuration upstream.

It's totally plausible that the regex module happens to avoid all these
issues. But you can't assume that it's a general solution.


Agreed. That was essentially what I was asking - has someone found a
solution to the issue of avoiding mixed CRTs when using mingw as a
compiler (or at least, found a better solution than using the original
mingw -lmsvcr100 approach combined with the distutils code that
supports this). Looks like the answer is that nobody has. But it's
certainly (still) true that for straightforward cases things seem
pretty much OK, especially if you avoid the "obvious" issues. (The
non-obvious ones sound terrifying, and I'm glad I don't have to write
code that encounters those!!!)


It's not *quite* "nobody has". Carl Kleffner has been doing a lot of
work on trying to set up Python-compatible gcc-based toolchains for
Windows. (This is still pretty much the only hope for getting a usable
free fortran compiler on Windows, and there's still lots of fortran
down at the bottom of the scientific/numerical stack). It's still
rather beta and the approach probably isn't sustainable without more
funding, but right now it's probably the closest thing you can get to
what you're asking about. I believe these are the latest builds:

https://binstar.org/carlkl/mingwpy/files

(Yes, those are .whl files that contain compiler toolchains.)

(Also, if making this *actually* work reliably would make a difference
to anyone, then feel free to send me heartfelt testimonials and I'll
forward them on to MS. Or I guess you could just send them to Steve
Dower directly ;-).)

-n



For anyone interested although this http://bugs.python.org/issue3871 
mingw issue is closed, it points to four other issues.  In total there 
are around 25 mingw issues open.  Maybe some of the work done on them 
can be tied up with the work that Carl Kleffner has done?


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

Re: [Python-Dev] VS 2010 compiler

[Nathaniel Smith]

On Mon, Sep 28, 2015 at 2:55 PM, Paul Moore  wrote:
> On 28 September 2015 at 22:18, Nathaniel Smith  wrote:
>> On Sep 28, 2015 1:39 PM, "Paul Moore"  wrote:
>>>
>>> On 28 September 2015 at 21:18, MRAB  wrote:
>>> > Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5,
>>> > both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet
>>> > that wasn't due to a bug in the source code.
>>>
>>> Interesting. With Python 3.5, what CRT does the module use? With
>>> Python 3.4 and earlier, distutils arranged for mingw builds to link to
>>> the CRT that matched the CPython build,
>>
>> This doesn't sound right to me. Maybe if you're talking about the original
>> mingw, but that's been abandoned upstream for years. Nowadays everyone uses
>> the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way
>> to select a non-default CRT is to hack up the toolchain's source tree and
>> cross your fingers. It's definitely not a supported configuration upstream.
>> (Source: we've been talking to mingw-w64 upstream and MS about how to
>> potentially make building free software less painful on Windows.)
>
> Interesting, it's been a long time since I've done work on the mingw
> support, so I hadn't realised this had happened.
>
> I guess that means that nowadays all mingw-compiled extensions link to
> msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?

Right, linking to msvcrt.dll is the only standard/supported
configuration for upstream mingw-w64.

>>> but I don't think that
>>> happened for 3.5 (because mingw doesn't support ucrt yet). So you'd
>>> have a CRT that doesn't match the core. Obviously this isn't causing
>>> you issues, but I'm not sure if it could (CRT problems seem to be
>>> mostly about "might cause issues" type problems...)
>>
>> CRT issues are indeed tricky, because they only bite in certain
>> circumstances -- so long as you never pass a FILE* or a fileno across the
>> dll boundary, or call malloc in one module and free in the other, or ...
>> then it works fine. Except when it doesn't :-).
>>
>> Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87
>> fpu should be configured in extended precision mode or not. So simply
>> loading a mingw-w64-compiled dll can potentially cause math functions in
>> other dlls to start returning incorrect results. Or maybe not -- it depends
>> on the details of how MS implemented things, and we can't exactly check. Or
>> whether you get correct math result in mingw-w64 compiled code can depend on
>> which runtime was used to spawn the thread that's running the code (!!). And
>> even if things work today, it could change without warning in any update,
>> because no one is testing this configuration upstream.
>>
>> It's totally plausible that the regex module happens to avoid all these
>> issues. But you can't assume that it's a general solution.
>
> Agreed. That was essentially what I was asking - has someone found a
> solution to the issue of avoiding mixed CRTs when using mingw as a
> compiler (or at least, found a better solution than using the original
> mingw -lmsvcr100 approach combined with the distutils code that
> supports this). Looks like the answer is that nobody has. But it's
> certainly (still) true that for straightforward cases things seem
> pretty much OK, especially if you avoid the "obvious" issues. (The
> non-obvious ones sound terrifying, and I'm glad I don't have to write
> code that encounters those!!!)

It's not *quite* "nobody has". Carl Kleffner has been doing a lot of
work on trying to set up Python-compatible gcc-based toolchains for
Windows. (This is still pretty much the only hope for getting a usable
free fortran compiler on Windows, and there's still lots of fortran
down at the bottom of the scientific/numerical stack). It's still
rather beta and the approach probably isn't sustainable without more
funding, but right now it's probably the closest thing you can get to
what you're asking about. I believe these are the latest builds:

https://binstar.org/carlkl/mingwpy/files

(Yes, those are .whl files that contain compiler toolchains.)

(Also, if making this *actually* work reliably would make a difference
to anyone, then feel free to send me heartfelt testimonials and I'll
forward them on to MS. Or I guess you could just send them to Steve
Dower directly ;-).)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Nathaniel Smith]

On Mon, Sep 28, 2015 at 5:14 PM, Mark Lawrence  wrote:
>
> For anyone interested although this http://bugs.python.org/issue3871 mingw
> issue is closed, it points to four other issues.  In total there are around
> 25 mingw issues open.  Maybe some of the work done on them can be tied up
> with the work that Carl Kleffner has done?

There are two different topics here: building extensions with
mingw-w64, and building CPython itself with mingw-w64. That issue is
about the latter, and I think everyone is agreed at this point that
Python.org isn't going to support that so long as there are
incompatible CRT/ABIs between mingw-w64 and MSVC, because that creates
all kinds of problems (e.g. now you have to distinguish between wheels
targeting win32-using-msvc versus win32-using-mingw-w64). So the first
thing to solve is to get the mingw-w64 CRT/ABI incompatibilities
sorted out and extensions working, and then once that's solid we could
think about whether CPython upstream is interested in carrying
mingw-w64 build patches for CPython itself.

(It might not even be that interesting at that point... the main
reason for wanting mingw-w64-built CPython is so that it can
interoperate with the mingw-w64 ecosystem -- e.g. supporting Python
plugins in mingw-w64's gdb. If MSVC and mingw-w64 python become
compatible with each other, then this becomes much less urgent...)

The msys2 folks who are maintaining the CPython-on-mingw-w64 patch set
are definitely aware of Carl's work.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[INADA Naoki]

You can use "Windows SDK for Windows 7 and .NET Framework 4".

http://www.microsoft.com/en-us/download/details.aspx?id=8279

On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

> As I understand it, the MS VS2010 compiler is required (or at least
> best practice) for compiling Python extensions for the python.org
> Windows builds of py 3.4 and ?[1]
>
> However, MS now makes it very hard (impossible?) to download VS2010
> Express ( or Community, or whatever the free as in beer version is
> called).
>
> I realize that this is not python-dev's responsibility, but if there
> is any way to either document where it can be found, or put a bit of
> pressure on MS to make it available, as they have for VS2008 and
> py2.7, that would be great.
>
> Sorry to bug this list, I didn't know where else to reach out to.
>
> -Chris
>
> [1] it's actually prefer hard to find out which compiler version is
> used for which python version. And has been for years. Would a patch
> to the docs, probably here:
>
> https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
>
> Be considered?
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>



-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Steve Dower]

On 25Sep2015 0824, Chris Barker - NOAA Federal wrote:

As I understand it, the MS VS2010 compiler is required (or at least
best practice) for compiling Python extensions for the python.org
Windows builds of py 3.4 and ?[1]

However, MS now makes it very hard (impossible?) to download VS2010
Express ( or Community, or whatever the free as in beer version is
called).


This is the main motivation for moving to VS 2015 for Python 3.5. See 
https://docs.python.org/3.5/using/windows.html#compiling-python-on-windows 
(same link as [1] with the version changed) and also 
https://docs.python.org/3.5/whatsnew/3.5.html



I realize that this is not python-dev's responsibility, but if there
is any way to either document where it can be found, or put a bit of
pressure on MS to make it available, as they have for VS2008 and
py2.7, that would be great.


I've applied that pressure already and it's not going to happen :(

The best "solution" is to move to Py 3.5 asap. If you're writing an 
extension and are being careful, you can use a later compiler to build 
for 3.4 with Py_LIMITED_API set and a lot of care (did I mention you 
need to be careful? It's not just FILE* that will break...).


Cheers,
Steve


Sorry to bug this list, I didn't know where else to reach out to.

-Chris

[1] it's actually prefer hard to find out which compiler version is
used for which python version. And has been for years. Would a patch
to the docs, probably here:

https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows

Be considered?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker - NOAA Federal]

> This is the main motivation for moving to VS 2015 for Python 3.5.

Yes,I've followed that work to the degree I understand it, and it
looks like a great move. Thanks so much for your (continuing) efforts
with this.

> put a bit of
>>
>> pressure on MS to make it available, as they have for VS2008 and
>> py2.7, that would be great.
>
> I've applied that pressure already and it's not going to happen :(

Darn. Makes no sense, but I work for almost as big a  bureaucracy as
MS, so I know how that goes... :-)

> The best "solution" is to move to Py 3.5 asap.

Really not a solution at all, I'm afraid, and there are still growing
pains with compiling for it as well (see Numpy/SciPy).

But thanks.

-Chris
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] VS 2010 compiler

[Chris Barker - NOAA Federal]

You can use "Windows SDK for Windows 7 and .NET Framework 4".

http://www.microsoft.com/en-us/download/details.aspx?id=8279


Thanks. Last time I tried that route, it was for 64 bit py2.7. And it
required some kludging of environment variables, and registry acces I don't
have permission for.

But it still may be the best bet. I'll give it a try when I have a chance.

And this should be in the "official" docs...

-CHB



On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

> As I understand it, the MS VS2010 compiler is required (or at least
> best practice) for compiling Python extensions for the python.org
> Windows builds of py 3.4 and ?[1]
>
> However, MS now makes it very hard (impossible?) to download VS2010
> Express ( or Community, or whatever the free as in beer version is
> called).
>
> I realize that this is not python-dev's responsibility, but if there
> is any way to either document where it can be found, or put a bit of
> pressure on MS to make it available, as they have for VS2008 and
> py2.7, that would be great.
>
> Sorry to bug this list, I didn't know where else to reach out to.
>
> -Chris
>
> [1] it's actually prefer hard to find out which compiler version is
> used for which python version. And has been for years. Would a patch
> to the docs, probably here:
>
> https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
>
> Be considered?
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>



-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com