Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
Zitat von Paul Moore p.f.mo...@gmail.com: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. Can you please be more specific? What SDK, and what free compilers? The bug report is about VS Express, not the SDK compilers. Two questions, really: * Is there any intention that building extensions with the SDK compilers is supported? As long as the SDK does include compilers: yes. * How do I do it, if so? Open a command window with the SDK environment variables set, then also set DISTUTILS_USE_SDK, and invoke setup.py As an alternative, I wonder whether Microsoft would be willing to support Appveyor by providing them with access to the full version of MSVC (2008 and 2010) for the build workers? Steve - do you know if there's any possibility of something like that? Not sure who is they and the build workers. If you are talking about the Python core developers - we can already have MSDN access if we want. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 07:14, mar...@v.loewis.de wrote: Zitat von Paul Moore p.f.mo...@gmail.com: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. Can you please be more specific? What SDK, and what free compilers? The bug report is about VS Express, not the SDK compilers. For Python 2.7, I was using the Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 x64 version. I set DISTUTILS_USE_SDK, and got exactly the symptom mentioned in the bug. Hence my comment that I was hitting the same issue with a different environment. I don't know whether it is relevant, but Visual Studio 2008 Express was also installed in the environment. Two questions, really: * Is there any intention that building extensions with the SDK compilers is supported? As long as the SDK does include compilers: yes. * How do I do it, if so? Open a command window with the SDK environment variables set, then also set DISTUTILS_USE_SDK, and invoke setup.py I'm scripting the build, which means I can't use the Open SDK command line environment start menu item. But what I do is SetEnv.cmd /x64 /release SET DISTUTILS_USE_SDK=1 SET MSSdk=1 I then run setup.py and get the error ValueError: [u'path'] I have not yet been able to successfully set up the SDK environment locally (the above was done on a remote machine hosted by appveyor.com) but am close to doing so. My normal build machine includes the full Visual Studio (via the MSDN access you mentioned). Once I have that environment I'll be in a position to reproduce the error on my local machine and do further testing much more conveniently. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
Hi Paul, I don't know how Olivier Grisel did it, but I can testify it does build extensions for Windows (32bit and 64bit) using appveyor. https://github.com/ogrisel/python-appveyor-demo/ Best wishes, Allen On Tue, Sep 23, 2014, at 06:42 PM, Paul Moore wrote: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. But I can't find any official documentation on how to do this, and everything I have tried so far has failed in frustrating ways. I'm now at the point where I appear to be hitting the following bug - http://bugs.python.org/issue7511 which has stumped me completely. Sadly, as is typical with distutils issues, this one seems to have been round for years and there is little or no sign that anyone is willing to fix it. Two questions, really: * Is there any intention that building extensions with the SDK compilers is supported? * How do I do it, if so? Personally, this is of limited relevance, as I have the full version of MSVC available. But I'm trying to put together some documentation for package developers on how to build Windows wheels, in particular using Appveyor to automate the process, with the intention that people shouldn't have to jump through hoops to provide wheels, but should rather be able to simply use a prebuilt recipe to automate the process. As an alternative, I wonder whether Microsoft would be willing to support Appveyor by providing them with access to the full version of MSVC (2008 and 2010) for the build workers? Steve - do you know if there's any possibility of something like that? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 9/23/2014 5:42 PM, Paul Moore wrote: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. But I can't find any official documentation on how to do this, and everything I have tried so far has failed in frustrating ways. I'm now at the point where I appear to be hitting the following bug - http://bugs.python.org/issue7511 which has stumped me completely. Sadly, as is typical with distutils issues, this one seems to have been round for years and there is little or no sign that anyone is willing to fix it. Two questions, really: * Is there any intention that building extensions with the SDK compilers is supported? * How do I do it, if so? Personally, this is of limited relevance, as I have the full version of MSVC available. But I'm trying to put together some documentation for package developers on how to build Windows wheels, in particular using Appveyor to automate the process, with the intention that people shouldn't have to jump through hoops to provide wheels, but should rather be able to simply use a prebuilt recipe to automate the process. As an alternative, I wonder whether Microsoft would be willing to support Appveyor by providing them with access to the full version of MSVC (2008 and 2010) for the build workers? Steve - do you know if there's any possibility of something like that? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Paul, Some of us from the Scientific Python side of development have been using appveyor to build Windows wheels for a few projects. A demo from one of developers of scikit-learn gives a good overview of the process we have been using [1]. The Cython wiki also has some information on getting the Windows SDK set up correctly for 64-bit compiling [2]. Personally I was able to get the pacakges I was working on to compile on a Windows host using only the Windows SDK compilers following the hints available on those to links and a Stack overflow answer on the topic [3]. It has been a few months since then but I can try to reproduce my work if those links don't provide the answers. Cheers, - Jonathan Helmus [1] https://github.com/ogrisel/python-appveyor-demo [2] https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows [3] http://stackoverflow.com/questions/11267463/compiling-python-modules-on-win-x64/13751649#13751649 ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 03:45, Jonathan J. Helmus jjhel...@gmail.com wrote: Some of us from the Scientific Python side of development have been using appveyor to build Windows wheels for a few projects. A demo from one of developers of scikit-learn gives a good overview of the process we have been using [1]. Thanks for the pointer. (Also thanks to Allen Riddell). I'll take a look. Ideally, what I'd like to do is write something up to help non-Windows experts get things up and running, so this will be very useful. Cheers. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On Wed, Sep 24, 2014 at 6:55 AM, Paul Moore p.f.mo...@gmail.com wrote: Thanks for the pointer. (Also thanks to Allen Riddell). I'll take a look. Ideally, what I'd like to do is write something up to help non-Windows experts get things up and running, so this will be very useful. Thanks -- that would be great. But really, why is this so hard? Win64 is essentially One platform, and the freely available SDK is ONE compiler environment. surely it's possible to write a batch script of some sort that you could put somewhere (or even deliver with python! ) so this would be: 1) download and install THIS (the sdk from MS) 2) run: set_up_win_complier.py 3) build the package: python setup.py build without needing to do multiple step, without needing to be in the special set-up command Window, etc. In fact, even better would be for distutils to run the mythical set_up_win_complier.py script for you. distutils does work out of the box with the VS2008 Express for 32 bit -- I'm still confused why this is so much harder for 64 bit. *sigh* -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 17:24, Chris Barker chris.bar...@noaa.gov wrote: Thanks -- that would be great. But really, why is this so hard? Win64 is essentially One platform, and the freely available SDK is ONE compiler environment. If only that were true :-) What I've found is: 1. Different SDKs are needed for Python 2.7 and 3.3+ (the VS2008/VS2010 split) 2. The v7.0 SDK (Python 2.7) is a bit of a beast to install correctly - I managed to trash a VM by installing the x86 one when I should have installed the x64 one. 3. There are bugs in the SDK - the setenv script for v7.0 needs fixes or it fails. Agreed, it should be easy. And indeed, it is if you have the full Visual Studio. But when Python 2.7 came out, the freely available MS tools were distinctly less convenient to use, and that shows. It's getting a lot better, and once we start using MSVC 2012 or later (i.e., Python 3.5+), the express editions include 64-bit support out of the box, which makes most of the problems go away. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 20:02, Chris Barker chris.bar...@noaa.gov wrote: t's getting a lot better, and once we start using MSVC 2012 or later (i.e., Python 3.5+), the express editions include 64-bit support out of the box, which makes most of the problems go away. Sure, but is there something we can do with the old stuff -- some of us will be ruing 2.7 for a good while yet! Steve wrote: As I mentioned at the start of this thread - hold your frustration and wait for a little while :) It wasn't clear -- will things get better for 2.7 ? OR just the new stuff? i.e. frustration aside, should I not bother to wrangle this now for my projects if I can hold off a bit? I can't speak for Steve, but personally, I do intend to work through these issues and write up a how to set things up to build wheels for your Python projects (probably using appveyor, as it has the environment already there, but likely also for a local setup) document. It'll be far from the first such document, but I'd like to see my version published under the PyPA banner and as such have the status of the official answer. My advice would be not to rush. If the currently available information is enough for you, by all means go for it, but if you're hitting difficulties (or just don't want to risk doing so) I'm hoping things will be improved[1] in the relatively short term, so it might be worth waiting. Paul [1] Where it's possible the only improvement is that you've got me as a specific target for your complaints about the lousy documentation, but I'm hoping I can do a *bit* better than that :-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On Wed, Sep 24, 2014 at 11:49 AM, Paul Moore p.f.mo...@gmail.com wrote: essentially One platform, and the freely available SDK is ONE compiler environment. If only that were true :-) What I've found is: 1. Different SDKs are needed for Python 2.7 and 3.3+ (the VS2008/VS2010 split) well, yeah, but that's not the problem at hand -- that one is ugly and painful and always has been :-( 2. The v7.0 SDK (Python 2.7) is a bit of a beast to install correctly - I managed to trash a VM by installing the x86 one when I should have installed the x64 one. Ah, what fun -- though if you DO install the right one, hopefully it will work, at least if it's installed with defaults, which most folks can do. 3. There are bugs in the SDK - the setenv script for v7.0 needs fixes or it fails. OK -- that sucks and is simply going make this painful -- darn it! Agreed, it should be easy. And indeed, it is if you have the full Visual Studio. But when Python 2.7 came out, the freely available MS tools were distinctly less convenient to use, and that shows. and they still are, too. It's getting a lot better, and once we start using MSVC 2012 or later (i.e., Python 3.5+), the express editions include 64-bit support out of the box, which makes most of the problems go away. Sure, but is there something we can do with the old stuff -- some of us will be ruing 2.7 for a good while yet! Steve wrote: As I mentioned at the start of this thread - hold your frustration and wait for a little while :) It wasn't clear -- will things get better for 2.7 ? OR just the new stuff? i.e. frustration aside, should I not bother to wrangle this now for my projects if I can hold off a bit? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
Chris Barker wrote: On Wed, Sep 24, 2014 at 6:55 AM, Paul Moore p.f.mo...@gmail.com wrote: Thanks for the pointer. (Also thanks to Allen Riddell). I'll take a look. Ideally, what I'd like to do is write something up to help non-Windows experts get things up and running, so this will be very useful. Thanks -- that would be great. But really, why is this so hard? Win64 is essentially One platform, and the freely available SDK is ONE compiler environment. surely it's possible to write a batch script of some sort that you could put somewhere (or even deliver with python! ) so this would be: 1) download and install THIS (the sdk from MS) 2) run: set_up_win_complier.py 3) build the package: python setup.py build without needing to do multiple step, without needing to be in the special set-up command Window, etc. In fact, even better would be for distutils to run the mythical set_up_win_complier.py script for you. distutils does work out of the box with the VS2008 Express for 32 bit -- I'm still confused why this is so much harder for 64 bit. Someone made a decision back when that express edition was released that people who _needed_ 64-bit compilers could justify paying for them. At the time (pre-Windows 7, which was the first usable 64-bit Windows), this made sense, but the world has changed since then and so have the later versions of VC++ Express/Express for Desktop, which now include all the compilers. *sigh* As I mentioned at the start of this thread - hold your frustration and wait for a little while :) Cheers, Steve -Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
Am 24.09.14 08:41, schrieb Paul Moore: On 24 September 2014 07:14, mar...@v.loewis.de wrote: Zitat von Paul Moore p.f.mo...@gmail.com: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. Can you please be more specific? What SDK, and what free compilers? The bug report is about VS Express, not the SDK compilers. For Python 2.7, I was using the Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 x64 version. I set DISTUTILS_USE_SDK, and got exactly the symptom mentioned in the bug. So what is the value of your vcvarsall.bat? Why could it not find the other interesting variables? If it's really the same issue: does any of the proposed patches help? Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 21:04, Martin v. Löwis mar...@v.loewis.de wrote: Am 24.09.14 08:41, schrieb Paul Moore: On 24 September 2014 07:14, mar...@v.loewis.de wrote: Zitat von Paul Moore p.f.mo...@gmail.com: Can anyone give me some advice, please? I am trying to build extensions on Windows 64-bit, using the free Windows SDK compilers. Can you please be more specific? What SDK, and what free compilers? The bug report is about VS Express, not the SDK compilers. For Python 2.7, I was using the Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 x64 version. I set DISTUTILS_USE_SDK, and got exactly the symptom mentioned in the bug. So what is the value of your vcvarsall.bat? Why could it not find the other interesting variables? I'm using setenv.cmd, not vcvarsall.bat (because that's the advice I found in the scattered documents I found). I'm not even sure I have a vcvarsall.bat that I can call (the only working environment I currently have access to with a SDK installed is only accessible in a convoluted manner which makes investigation painful (appveyor, if you know the system) If it's really the same issue: does any of the proposed patches help? I am still in the process of trying to get a usable local environment with the SDK installed. Once I do, I'll report back. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 03:45, Jonathan J. Helmus jjhel...@gmail.com wrote: Some of us from the Scientific Python side of development have been using appveyor to build Windows wheels for a few projects. A demo from one of developers of scikit-learn gives a good overview of the process we have been using [1]. This is excellent. Many thanks for the pointer - you've clearly managed to solve some of the more annoying problems that I have been hitting. (I'd claim that I was getting there, but you've saved me the effort :-)) One thing I have done is request the Appveyor team to add 64-bit Pythons to their build environments, which they have done, so that now there should be no need to install your own copy of Python (at least for 2.7, 3.3 and 3.4). I've copied Olivier in here as the author of the demo project, but would you mind if I used this as the basis of a document covering how to build wheels for your project using Appveyor? Obviously, I'd give you full credit. I'm thinking of including it as a section in the Python packaging guide, or maybe as a separate HOWTO document. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers
On 24 September 2014 22:58, Olivier Grisel olivier.gri...@ensta.org wrote: Under which path? It's now documented in http://www.appveyor.com/docs/installed-software, but C:\PythonXY and C:\PythonXY-x64. Could you please issue a PR to: https://github.com/ogrisel/python-appveyor-demo to show how to leverage pre-installed versions of Python? Will do (although it might be a few days, I'm pretty snowed under at work right now). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
