Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-12 Thread Ray Donnelly 
On Sun, Oct 12, 2014 at 9:43 AM, Roumen Petrov
 wrote:
> Victor Stinner wrote:
>>
>> Hi,
>
> [SKIP]
>>
>> === MinGW
>>
>> Some people tried to compile Python. See for example:
>> https://bitbucket.org/puqing/python-mingw
>>
>> We even got some patches:
>> http://bugs.python.org/issue3871 (rejected)
>
> [SNIP]
>
> As "all in one" patch it was rejected , but you could find splits:
> 17605 - mingw-meta: build interpeter core
> 18653 - mingw-meta: build core modules
>
> Lot of people post links to possible issues using GCC windows compiler. A
> lot of them are not real issues for CPython.
>
>
> In addition for those why would like to cross compile C-extensions for MS
> Windows either from linux of cygwin then could use this set:
> 18654 - modernize mingw&cygwin compiler classes
>
>
> I could step in as maintainer for Cygwin and builds based on GCC using
> mingw* APIs.
>

+1 for Roumen maintaining GCC cross builds using mingw*.

As Rafael Villar Burke mentioned, the MSYS2 project has Native Windows
Python builds (for both 3.4.2 and 2.7.8). We use Roumen's split
patches (and then our own on top):
https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python3
and https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python2.

To install MSYS2, build then test 32bit and 64bit mingw-w64-python3 on
a fresh 64bit Windows installation:
Download 
http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download
Run msys2-x86_64-20141003.exe and install to a (short) path without
spaces or non-ascii characters (C:\msys64 is good), keep "Run MSYS2
64bit now." ticked. The remaining commands are to be entered in the
MSYS2 mintty shell.
# Install the packages necessary to build mingw-w64-python*
#  using Pacman package manager (answer Y or press enter when prompted):
pacman -S base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain
# Download the source recipes and patches
#  that are used to build all of MSYS2's mingw-w64 packages:
git clone https://github.com/Alexpux/MINGW-packages
cd MINGW-packages/mingw-w64-python3
# Build it:
# (s == sync (install) necessary {make,}dependencies
#  L == write log files)
# answer Y or press enter when prompted
# (remove --nocheck if you want to run the testsuite before packaging)
makepkg-mingw -sL --nocheck

# To install the newly built packages:
pacman -U mingw-w64-*.xz

# To run them, you should add /mingw64/bin or /mingw32/bin to your PATH
# (or launch a new shell via mingw32_shell.bat or mingw64_shell.bat)
# Of course, if you don't want to build it from source you can simply issue:
pacman -S mingw-w64-python3

.. all of the above applies equally to mingw-w64-python2.

If anyone would like to help us to get our work into shape and then
merged we would be extremely grateful. Unfortunately Python is one of
our most patched packages.

In response to Steve Dower's request for discussion:

Having an alternative, fully Open Source build system for Python on
Windows using a stable Win32 ABI which is compatible all the way back
to Windows XP SP3 and Windows XP 64 and can interoperate out of the
box with many other tools and libraries (numpy, GNU Fortran - yes we
have this, pyQt, pyGTK etc) is something many people would dearly
like. We don't wish to usurp Visual Studio as the recommended build
system for Windows, we simply want to enable an Open Source choice.
For philosophical and practical reasons there are many people who wish
to limit their exposure to proprietary, closed build tools such as
Visual Studio. That Windows has always been a much more difficult
platform for Open Source development is not something that the Open
Source community should accept and then work around, rather something
we should try to fix. For information on contributing to MSYS2 please
see https://sourceforge.net/p/msys2/wiki/Contributing%20to%20MSYS2/

Finally, this thread has contained many references to mingw, care
should be taken to be explicit about which of MinGW-w64 or mingw is
being referred to, since they are two different projects. MinGW-w64
supports 64bit and a lot of work is being done to support ARM.

Best regards,

Ray Donnelly.

>
> Regards,
> Roumen Petrov
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Ray Donnelly 
On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower  wrote:
> (Apologies for the short reply, posting from my phone.)
>
> "MSVC can continue
> to be the default compiler used for Python on Windows, none of Roumen's
> patches change that. They would merely open up the choice for packagers and
> users to build CPython (and extension modules, thanks to separate patches)
> with alternate compilers, in cross-compilation or otherwise."
>
> Building CPython for Windows is not something that needs solving. The
> culture on Windows is to redistribute binaries, not source, and both the
> core team and a number of redistributors have this figured out (and it will
> only become easier with VC14 and Python 3.5).

This is the second time you've used the vacuous "culture on Windows"
argument, now with an added appeal to (vague) authority. That may be
your opinion and that of some others, but there's a large number of
people who don't care for using non-Free tools. IMHO building CPython
on Windows using Open Source toolchains is very much something that
needs merging upstream and supporting by default. What is it that you
are afraid of if CPython can be compiled out of the box using
mingw/MinGW-w64? Why are you fighting so hard against having option.
If CPython wants to truly call itself an Open Source project then I
consider being able to compile and cross-compile it with capable Open
Source toolchains on all major platforms a requirement.

>
> I'd rather see this effort thrown behind compiling extensions, including
> cross compilation. The ABI is well defined enough that any compiler should
> be usable, especially once the new CRT is in use. However, there is work
> needed to update the various tool chains to link to VC14's CRT and we need
> to figure out the inconsistencies between tools so we can document and work
> through them.
>
> Having different builds of CPython out there will only fragment the
> community and hurt extension authors far more than it may seem to help.
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> 
> From: Tony Kelman
> Sent: ‎10/‎25/‎2014 9:06
> To: python-dev@python.org
> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
>
> I'm several weeks late to this discussion, but I'm glad to see that it
> happened. I'm not a Python developer, and barely a user, but I have several
> years of daily experience compiling complicated scientific software cross-
> platform, particularly with MinGW-w64 for Windows. The Python community,
> both core language and scientific package developers and users, needs to
> act here. The problem is bad and getting worse. Luckily much of the work
> to start solving it has already been done in bits and pieces, it needs
> coordination and participation to come to a conclusion.
>
>> Cross compilation is a valid issue, but I hope that build services like
>> Appveyor also help out here. There is regular talk about the PSF/PyPI
>> providing something similar
>
> AppVeyor is better than nothing (I've been using it since beta), but it's
> a far cry from build services and package management as the Linux world
> knows them. Obtaining and setting up build dependencies quickly and
> repeatably, and finishing the build of a complicated piece of software
> such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
> lies), etc. on a small single-core VM with limited memory and a restrictive
> time limit is often not possible. These problems are solved within Linux
> infrastructure like Koji, Open Build Service, buildd, etc.
>
> MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
> compiling a wide variety of libraries from Linux to Windows, in addition to
> building conventionally on Windows for Windows. The MSYS2 collection of
> MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
> been mentioned. Linux distributions including
> - Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
> - openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
> - Arch https://aur.archlinux.org/packages/?K=mingw
> and others have projects for providing many hundreds of open-source
> packages compiled for Windows. Debian has the cross-compilers available but
> not many packages yet (https://packages.debian.org/search?keywords=mingw).
>
> As a developer of a (compiled) open-source library or application, wouldn't
> you love to be able to build binaries on Linux for Windows? With some work
> and build system patches, you can. For many projects it's a simple matter of
> ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
> only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
> Arch. This is possible with a very, very long set of patches, many of which
> have been submitted by Roumen Petrov to the Python bug tracker - see
> http://bugs.python.org/issue17605 and other issues linked therein. Roumen
> has done a huge amount o

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Ray Donnelly 
On Sat, Oct 25, 2014 at 10:52 PM, Antoine Pitrou  wrote:
> On Sat, 25 Oct 2014 21:10:23 +0100
> Ray Donnelly  wrote:
>>
>> This is the second time you've used the vacuous "culture on Windows"
>> argument, now with an added appeal to (vague) authority.
> [...]
>> Why are you fighting so hard against having option.
>> If CPython wants to truly call itself an Open Source project then I
>> consider being able to compile and cross-compile it with capable Open
>> Source toolchains on all major platforms a requirement.
>
> Now *that* sounds vacuous.
>

Maybe you missed where I said "I consider".

> Regarding open source, there's a clear and official definition of it,
> which Python satisfies:
> http://opensource.org/osd-annotated
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 12:30 AM, Antoine Pitrou  wrote:
> On Sat, 25 Oct 2014 19:24:38 -0400
> "R. David Murray"  wrote:
>>
>> I know I for one do not generally test patches on Windows because I
>> haven't taken the time to learn how to build CPython on it.  Sure, I
>> could test pure python changes by applying patches to an installed
>> Python, but that's an ongoing pain and I'd rather learn to build CPython
>> on Windows and get to use the normal hg tools.
>>
>> If I could use a more linux-like toolchain to build CPython on windows,
>
> Well, I don't know how "linux-like" you want your toolchain, but FTR
> you should be able to build from the command line by running
> "Tools\buildbot\build.bat".
>
> I doubt the MinGW case can be much simpler :-)
>

I beg to differ:

"Tools\buildbot\build.bat" contains:
@rem Used by the buildbot "compile" step.
cmd /c Tools\buildbot\external.bat
call "%VS100COMNTOOLS%vsvars32.bat"
cmd /c Tools\buildbot\clean.bat
msbuild PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32

^ this involves purchasing and installing MS Visual Studio (I'm not
sure if the Express Edition can be used).

Without explanations for what each step is doing (I posted those last
week), on MSYS2:
Download and run:
http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download
>From the MSYS2 shell:
pacman -S git base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain
git clone https://github.com/Alexpux/MINGW-packages
cd MINGW-packages/mingw-w64-python3
makepkg-mingw -sL --nocheck
(remove --nocheck to run the testsuite. Also you could put this into a
batch file if you were so inclined.)

To install the newly built packages, from the MSYS2 shell again:
pacman -U mingw-w64-*.xz

To run them, you should add /mingw64/bin or /mingw32/bin to your PATH
(or launch a new shell via mingw32_shell.bat or mingw64_shell.bat)
Of course, if you don't want to build it from source you can simply issue:
pacman -S mingw-w64-python3

.. all of the above applies equally to mingw-w64-python2.

> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Ray Donnelly 
On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
> On 25 October 2014 23:22, Chris Angelico  wrote:
>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
>>> My point is that your "Windows build" would not have the same behaviour
>>> as a MSVC-produced Windows build, and so testing it with it would not
>>> certify that your code would actually be compatible with genuine
>>> MSVC builds of CPython, which we will not stop supporting.
>>>
>>
>> So you're saying it's impossible to support two compilers?
>
> No, rather that Windows currently only has a single supported compiler
> (excluding cygwin, which is essentially a different OS). Adding a
> second compiler doesn't just involve adding support for it - which is
> all that the people offering mingw patches are doing - but also
> involves going through the whole CPython ecosystem locating the places
> where there is an implicit assumption that "all Windows builds use the
> same compiler" and fixing them. I've already pointed out where this is
> a question for pip and wheel. Whoever wants to add support for a
> second compiler needs to be willing to do this part of the job as
> well.
>
> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.

The msvcrt.dlls that MinGW-w64 depends on are those dating back to
Windows XP SP3 / XP64. Ironically, the official Windows CPython
doesn't come with any such crt guarantees and you must ensure that the
same msvcr??.dll is used for *all* extensions. This puts considerable
strain on extension developers to use the correct (or any) version of
Visual Studio to build their extensions for CPython on Windows. Also,
where are the publicly accessible specifications and other technical
descriptions that MinGW-w64 would need to implement strong binary
compatibility with MSVC? As a random example, those for C++ name
mangling and the PDB file format would be very helpful.

>
> Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-25 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 1:45 AM, Steve Dower  wrote:
> Ray Donnelly wrote:
>> On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
>>> On 25 October 2014 23:22, Chris Angelico  wrote:
>>>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  
>>>> wrote:
>>>>> My point is that your "Windows build" would not have the same behaviour
>>>>> as a MSVC-produced Windows build, and so testing it with it would not
>>>>> certify that your code would actually be compatible with genuine
>>>>> MSVC builds of CPython, which we will not stop supporting.
>>>>>
>>>>
>>>> So you're saying it's impossible to support two compilers?
>>>
>>> No, rather that Windows currently only has a single supported compiler
>>> (excluding cygwin, which is essentially a different OS). Adding a
>>> second compiler doesn't just involve adding support for it - which is
>>> all that the people offering mingw patches are doing - but also
>>> involves going through the whole CPython ecosystem locating the places
>>> where there is an implicit assumption that "all Windows builds use the
>>> same compiler" and fixing them. I've already pointed out where this is
>>> a question for pip and wheel. Whoever wants to add support for a
>>> second compiler needs to be willing to do this part of the job as
>>> well.
>>>
>>> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.
>>
>> The msvcrt.dlls that MinGW-w64 depends on are those dating back to
>> Windows XP SP3 / XP64. Ironically, the official Windows CPython
>> doesn't come with any such crt guarantees and you must ensure that the
>> same msvcr??.dll is used for *all* extensions. This puts considerable
>> strain on extension developers to use the correct (or any) version of
>> Visual Studio to build their extensions for CPython on Windows.
>
> We're well aware of this, and it's a big part of why I'm currently migrating 
> CPython to build with VC14, which will not have the same binary compatibility 
> issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind 
> calls into DLLs, so provided the calling conventions match (which they must 
> or everything would crash very quickly), it should be relatively easy to 
> build compatible extensions with MinGW-w64.

Compatibility going forwards though, right? Still it's great to see
positive steps being made for the future of the Windows platform.

>
>> Also, where are the publicly accessible specifications and other technical
>> descriptions that MinGW-w64 would need to implement strong binary
>> compatibility with MSVC? As a random example, those for C++ name
>> mangling and the PDB file format would be very helpful.
>
> C++ name mangling is always an implementation detail and it changes from 
> version to version. Luckily, CPython is entirely in C, so that doesn't 
> matter. PDBs are another red herring - you can build a loadable PE file 
> without PDBs.

Of course C++ can be called from C and that is done in some CPython
extensions, so it's not a red herring. If we want to talk about strong
binary compatibility I'd expect the aim would be to intermix freely
between compilers. We'd like people to be able to debug MinGW-w64 code
using CDB in Visual Studio if they want to, and on the flipside, to
have GDB able to read PDB files built by MSVC (actually there's a long
standing problem when debugging MinGW-w64 code in GDB that stack
unwinding out of MS built dlls is flaky at best) - so again this is
not really a red herring. I'm also led to believe that MSVC has a very
good optimizer so if some project wanted to build certain libraries or
objects with that for their performance critical paths then I can see
that as being useful to those projects and their users'.

>
> The full source code for the MSVCRT is available with any version of Visual 
> Studio (including the free editions, last time I checked), so feel free to 
> check whatever you need to ensure compatibility. I've suggested to the VC 
> team that they could get in touch with the MinGW projects and offer to help 
> them improve compatibility with MSVC, but unfortunately I don't think anyone 
> will take me up on that. I'm happy to research what I can to answer specific 
> questions, but there's very little that isn't already publicly available 
> other than direct access to the devs.

Under what license? We'd rather have open specifications that
copyrighted, strictly licensed code that we can't look at for various
tainting reasons.

>
> Cheers,
> Steve
>
>>> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 1:12 PM, Tony Kelman  wrote:
> Thanks all for the responses. Clearly this is a subject about which
> people feel strongly, so that's good at least. David Murray's guidance
> in particular points to the most likely path to get improvements to
> really happen.
>
> Steve Dower:
>> Building CPython for Windows is not something that needs solving.
>
> Not in your opinion, but numerous packagers of MinGW-based native or
> cross-compiled package sets would love to include Python. The fact
> that they currently can't, without many patches, is a problem.
>
>> The culture on Windows is to redistribute binaries, not source,
>
> There are many cultures using Windows. Including open-source ones.
>
>> and both the core team and a number of redistributors have this figured
>> out (and it will only become easier with VC14 and Python 3.5).
>
> With MSVC. It doesn't work with MinGW, it likely doesn't work with Clang.
> MSVC is not the only compiler on Windows. There are many use cases for
> preferring other compilers. Have you read this wiki page for example?
> https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows
>
> In my personal experience, having recently gotten Julia to compile using
> MSVC for the first time, MSVC as a compiler is highly deficient for many
> needs especially in the scientific software community:
> - C99 (getting better recently, but still not done)
> - AT&T syntax assembly
> - C++11 features (also mostly okay now, but not if you're using an older
>   MSVC version with Python 2.7, which many people still have to do)
> - 128-bit integer intrinsics
> - cannot cross-compile from anything that isn't Windows
> - build systems foreign relative to shell/makefile systems used by most
>   open-source projects, few projects have time to maintain 2 separate build
>   systems (cmake helps but takes a lot of initial effort to convert to)
> - no free-as-in-beer Fortran compiler available
>
> I have none of these problems when I use MinGW-w64. Hence the desire to
> be able to curate an all-MinGW software stack. It's not a matter of open-
> source ideology for me, it's brass tacks "can I do the work I need to do."
> With MSVC I can't, with MinGW-w64 I can. Not being able to include CPython
> in an all-MinGW stack hurts, a lot.
>
> Only cross-compilation and the build system in the above list are relevant
> to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
> real reasons for some groups of users and developers to prefer MinGW-w64
> over MSVC.
>
>> I'd rather see this effort thrown behind compiling extensions,
>> including cross compilation.
>
> There are patches awaiting review that improve this as well. Efforts to
> improve CPython's build system and the handling of extensions are not
> completely independent, in many cases the patches are written by the same
> set of MinGW users. One of these sets of patches is not inherently evil,
> you understandably have less interest in them but it's still disappointing
> to see so little movement on either.
>
>> Having different builds of CPython out there will only fragment the
>> community and hurt extension authors far more than it may seem to help.
>
> The community of people developing and using open-source projects, either
> CPython or otherwise, is already highly fragmented. Ignoring it makes it
> worse. python.org does not have to distribute or endorse MinGW-compiled
> builds of CPython. If the build option never gets incorporated, then it
> will continue to be reverse-engineered.
>
> Guido van Rossum:
>> Here's the crux of the matter. We want compiled extension modules
>> distributed via PyPI to work with the binaries distributed from
>> python.org.
>
> Absolutely. I don't think additional options in the build system would
> change this.
>
> R. David Murray:
>> And, at this point, we would NEED A BUILDBOT.  That is, a machine that
>> has whatever tools are required installed such that tests added to the
>> test suite to test MinGW support in distutils would run, so we can be
>> sure we don't break anything when making other changes.
>
> That's not too hard. I've done this for other projects. AppVeyor works if
> your build is short enough, and I've done cross-compilation from Travis
> CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
> infrastructure, but I can offer guidance if it would help.
>
> Steve Dower:
>> I'm afraid of users having numpy crash because they're using an MSVC
>> CPython instead of a mingw CPython. I'm afraid of users not being able
>> to use library A and library B at the same time because A requires MSVC
>> CPython and B requires mingw CPython. (I can produce more examples if you
>> like, but the general concern is having a fragmented community, as I said
>> in my previous post.)
>
> A valid fear. Mixing C runtimes can cause problems, I've seen this myself.
> Correct me if I'm wrong, but this is nearly as much of an issue if someone
> wants to use a different versi

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 2:28 PM, Ray Donnelly  wrote:
> On Sun, Oct 26, 2014 at 1:12 PM, Tony Kelman  wrote:
>> Thanks all for the responses. Clearly this is a subject about which
>> people feel strongly, so that's good at least. David Murray's guidance
>> in particular points to the most likely path to get improvements to
>> really happen.
>>
>> Steve Dower:
>>> Building CPython for Windows is not something that needs solving.
>>
>> Not in your opinion, but numerous packagers of MinGW-based native or
>> cross-compiled package sets would love to include Python. The fact
>> that they currently can't, without many patches, is a problem.
>>
>>> The culture on Windows is to redistribute binaries, not source,
>>
>> There are many cultures using Windows. Including open-source ones.
>>
>>> and both the core team and a number of redistributors have this figured
>>> out (and it will only become easier with VC14 and Python 3.5).
>>
>> With MSVC. It doesn't work with MinGW, it likely doesn't work with Clang.
>> MSVC is not the only compiler on Windows. There are many use cases for
>> preferring other compilers. Have you read this wiki page for example?
>> https://github.com/numpy/numpy/wiki/Numerical-software-on-Windows
>>
>> In my personal experience, having recently gotten Julia to compile using
>> MSVC for the first time, MSVC as a compiler is highly deficient for many
>> needs especially in the scientific software community:
>> - C99 (getting better recently, but still not done)
>> - AT&T syntax assembly
>> - C++11 features (also mostly okay now, but not if you're using an older
>>   MSVC version with Python 2.7, which many people still have to do)
>> - 128-bit integer intrinsics
>> - cannot cross-compile from anything that isn't Windows
>> - build systems foreign relative to shell/makefile systems used by most
>>   open-source projects, few projects have time to maintain 2 separate build
>>   systems (cmake helps but takes a lot of initial effort to convert to)
>> - no free-as-in-beer Fortran compiler available
>>
>> I have none of these problems when I use MinGW-w64. Hence the desire to
>> be able to curate an all-MinGW software stack. It's not a matter of open-
>> source ideology for me, it's brass tacks "can I do the work I need to do."
>> With MSVC I can't, with MinGW-w64 I can. Not being able to include CPython
>> in an all-MinGW stack hurts, a lot.
>>
>> Only cross-compilation and the build system in the above list are relevant
>> to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
>> real reasons for some groups of users and developers to prefer MinGW-w64
>> over MSVC.
>>
>>> I'd rather see this effort thrown behind compiling extensions,
>>> including cross compilation.
>>
>> There are patches awaiting review that improve this as well. Efforts to
>> improve CPython's build system and the handling of extensions are not
>> completely independent, in many cases the patches are written by the same
>> set of MinGW users. One of these sets of patches is not inherently evil,
>> you understandably have less interest in them but it's still disappointing
>> to see so little movement on either.
>>
>>> Having different builds of CPython out there will only fragment the
>>> community and hurt extension authors far more than it may seem to help.
>>
>> The community of people developing and using open-source projects, either
>> CPython or otherwise, is already highly fragmented. Ignoring it makes it
>> worse. python.org does not have to distribute or endorse MinGW-compiled
>> builds of CPython. If the build option never gets incorporated, then it
>> will continue to be reverse-engineered.
>>
>> Guido van Rossum:
>>> Here's the crux of the matter. We want compiled extension modules
>>> distributed via PyPI to work with the binaries distributed from
>>> python.org.
>>
>> Absolutely. I don't think additional options in the build system would
>> change this.
>>
>> R. David Murray:
>>> And, at this point, we would NEED A BUILDBOT.  That is, a machine that
>>> has whatever tools are required installed such that tests added to the
>>> test suite to test MinGW support in distutils would run, so we can be
>>> sure we don't break anything when making other changes.
>>
>> That's not too hard. I've done this for other projects. AppVeyor works if
>> your build is

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-26 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 10:41 PM, Paul Moore  wrote:
> On 26 October 2014 13:12, Tony Kelman  wrote:
>> Only cross-compilation and the build system in the above list are relevant
>> to CPython, but I hope I have convinced you, Paul Moore, etc. that there are
>> real reasons for some groups of users and developers to prefer MinGW-w64
>> over MSVC.
>
> Not really, to be honest. I still don't understand why anyone not
> directly involved in CPython development would need to build their own
> Python executable on Windows. Can you explain a single specific
> situation where installing and using the python.org executable is not
> possible (on the assumption that the mingw build is functionally
> identical and ABI compatible with the CPython build, the claim being
> made here)?

I don't know where this "ABI compatible" thing came into being; I
think Steve Dower eluded to it by stating that we should focus on
enabling MinGW-w64 as an extension-building compiler, using a core
interpreter built with MSVC, and that by limiting the interfaces to
the Windows C calling conventions everything would be OK.
Unfortunately this is not possible. MinGW-w64-built extensions need to
link to msvcrt.dll to do anything useful and you cannot mix two
different msvcr??.dlls in one application. Please see
http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.100%29.aspx
and http://msdn.microsoft.com/en-us/library/ms235460%28v=VS.100%29.aspx
for the details. MinGW-w64 assumes the very old msvcrt.dll files from
Windows XP SP3 and XP64 specifically to avoid this mess. The rest of
your reply assumes that this ABI compatibility is a given so I'll stop
at this point.

> Note that "not possible" is different from "I don't want
> to" or "it doesn't fit my views about free software" or similar. Also
> note that building extensions is different - you have to assume that
> building extensions using mingw with a mingw-built CPython is just as
> hard as building them with a MSVC-built CPython (otherwise you've made
> changes to extension building and you should contribute them
> independently so that everyone can benefit, not just those who build
> their own Python with mingw!)
>
>> Paul Moore:
>>> If it were possible to cross-compile compatible extensions on Linux,
>>> projects developed on Linux could supply Windows binaries much more
>>> easily, which would be a huge benefit to the whole Windows Python
>>> community.
>>
>> I want to do exactly this in an automated repeatable way, preferably on
>> a build service. This seems harder to do when CPython cannot itself be
>> built and handled as a dependency by that same automated, repeatable
>> build service.
>
> I cannot see why you would need to build Python in order to build
> extensions. After all, if your build service is on Linux, it couldn't
> run a mingw-built Python anyway. If your build service is a Windows
> machine, just install the python.org binaries (which is a simple
> download and install, that can be fully automated, but which is a
> one-off process anyway).
>
>> Unless it becomes possible to cross-compile extensions
>> using the build machine's own version of Python, which might be the right
>> approach.
>
> This may be where we are getting confused. I see this as the only
> practical way of cross-compiling Windows extensions on Linux, by using
> the Linux Python. So being able to cross-compile Python is not
> relevant.
>
> On a tangential note, any work on supporting mingw builds and
> cross-compilation should probably be done using setuptools, so that it
> is external to the CPython code. That way (a) it isn't constrained by
> the CPython release schedules and backward compatibility constraints,
> and (b) it can be used in older versions of Python (which is pretty
> much essential if it's to be useful, TBH).
>
> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Ray Donnelly 
On Sun, Oct 26, 2014 at 11:52 PM,   wrote:
>
> Zitat von Tony Kelman :
>
>> A maintainer has volunteered. Others will help. Can any core developers
>> please begin reviewing some of his patches?
>
>
> Unfortunately, every attempt to review these patches has failed for me,
> every time. In the last iteration of an attempt to add mingw64 support,
> I had asked contributors to also provide instructions on how to use these
> patches, and haven't received any instructions that actually worked.
>
> I'm hesitant to add code that I cannot verify as actually working.
>
> I guess anybody else reviewing these patches ran into similar problems
> (I know some other core developers have tried reviewing them as well,
> others have stated here that they are unable to review the patches).
>

https://mail.python.org/pipermail/python-dev/2014-October/136756.html

> Regards,
> Martin
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

2014-10-27 Thread Ray Donnelly 
On Mon, Oct 27, 2014 at 8:54 PM, Steve Dower  wrote:
> Greg Ewing wrote:
>> Nick Coghlan wrote:
>>> That assumption will allow MinGW-w64 to link with the appropriate
>>> MSVCRT versions for extention building without anything breaking.
>>
>> If that works, then the same technique should allow CPython itself to be 
>> built
>> in a VS-compatible way with mingw, shouldn't it?
>>
>> Those objecting to a mingw-built python seem to be assuming that such a thing
>> will necessarily be incompatible with VS builds, but I don't see why that 
>> has to
>> be the case.
>
> That's true, and a good point that I missed. However, the main (practical) 
> desire for building CPython with something other than VS seems to be to avoid 
> having to be compatible with VS.

I've no idea where you get that impression from, no one has expressed
anything even approximating that. For me it's to avoid using closed
source software for my hobbyist programming and to help to create a
vibrant Open Source distribution for Windows, because I quite like
Windows; it's got a lot going for it. For others it's to ensure that
everything they care about (CPython with Fortran for example) works
together properly and reliably. I expect that avoiding compatibility
couldn't be further from any of our wishes.

>
> It's entirely possible that having two alternative builds of CPython would 
> force everyone to be more compatible, but I think it's more likely to simply 
> end up being two different worlds. Maybe I'm being unnecessarily cynical :)
>
> Cheers,
> Steve
>
>> --
>> Greg
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

2018-01-30 Thread Ray Donnelly 
On Tue, Jan 30, 2018 at 5:42 PM, Chris Barker  wrote:
> Ned,
>
> It looks like you're still building OS-X the same way as in the past:
>
> Intel 32+64 bit, 10.6 compatibility
>
> Is that right?
>
> Might it be time for an update?
>
> Do we still need to support 32 bit?  From:
>
> https://apple.stackexchange.com/questions/99640/how-old-are-macs-that-cannot-run-64-bit-applications
>
> There has not been a 32 bit-only Mac sold since 2006, and a out-of the box
> 32 bit OS since 2006 or 2007
>
> I can't find out what the older OS version Apple supports, but I know my IT
> dept has been making me upgrade, so I"m going to guess 10.8 or newer...
>
> And maybe we could even get rid of the "Framework" builds..

While we're making such macOS-build requests, any chance of building a
static interpreter too? We've been doing that on the Anaconda
Distribution since the 5.0 release in September and it seems to be
working well.

>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

2018-01-31 Thread Ray Donnelly 
On Jan 30, 2018 6:47 PM, "Joni Orponen"  wrote:

On Tue, Jan 30, 2018 at 6:50 PM, Ray Donnelly 
wrote:

> While we're making such macOS-build requests, any chance of building a
> static interpreter too? We've been doing that on the Anaconda
> Distribution since the 5.0 release in September and it seems to be
> working well.
>

PyPy is also currently eyeing doing their macOS builds better:
https://bitbucket.org/pypy/pypy/issues/2734/establish-a-build-and-release-
pipeline-for

What do the Anaconda static builds get built on?


We have our own clang pseudo cross-compilers and use a macOS 10.9 SDK for
all of our package compilation to achieve compatibility (this means we can
compile on newer macOS just fine). We see a 1.1 to 1.2 times performance
benefit over official releases as measured using 'python performance'.

Apart from a static interpreter we also enable LTO and PGO and only build
for 64-bit so I'm not sure how much each bit continues. Our recipe for
python 3.6 can be found at:

https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe


-- 
Joni Orponen

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

2018-01-31 Thread Ray Donnelly 
On Jan 31, 2018 8:31 AM, "Ray Donnelly"  wrote:



On Jan 30, 2018 6:47 PM, "Joni Orponen"  wrote:

On Tue, Jan 30, 2018 at 6:50 PM, Ray Donnelly 
wrote:

> While we're making such macOS-build requests, any chance of building a
> static interpreter too? We've been doing that on the Anaconda
> Distribution since the 5.0 release in September and it seems to be
> working well.
>

PyPy is also currently eyeing doing their macOS builds better:
https://bitbucket.org/pypy/pypy/issues/2734/establis
h-a-build-and-release-pipeline-for

What do the Anaconda static builds get built on?


We have our own clang pseudo cross-compilers and use a macOS 10.9 SDK for
all of our package compilation to achieve compatibility (this means we can
compile on newer macOS just fine). We see a 1.1 to 1.2 times performance
benefit over official releases as measured using 'python performance'.

Apart from a static interpreter we also enable LTO and PGO and only build
for 64-bit so I'm not sure how much each bit continues. Our recipe for
python 3.6 can be found at:


s/continues/contributes/


https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe


-- 
Joni Orponen

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/mingw.
android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] OS-X builds for 3.7.0

2018-01-31 Thread Ray Donnelly 
On Wed, Jan 31, 2018 at 11:16 AM, Joni Orponen  wrote:
> On Wed, Jan 31, 2018 at 9:31 AM, Ray Donnelly 
> wrote:
>>
>> We see a 1.1 to 1.2 times performance benefit over official releases as
>> measured using 'python performance'.
>>
>> Apart from a static interpreter we also enable LTO and PGO and only build
>> for 64-bit so I'm not sure how much each bit continues. Our recipe for
>> python 3.6 can be found at:
>
>
> Do you metrify LTO and PGO independent of each other as well or only the
> "enable everything" combo? I've had mixed results with LTO so far, but this
> is probably hardware / compiler combination specific.

I've never found enough time to take detailed metrics, sorry. Maybe
one day? Looking at my performance graphs again:

Against the official CPython 3.6 (probably .3 or .4) release I see:
1 that is 2.01x faster (python-startup, 24.6ms down to 12.2ms)
5 that are >=1.5x,<1.6x faster.
13 that are >=1.4x,<1.5x faster.
21 that are >=1.3x,<1.4x faster.
14 that are >=1.2x,<1.3x faster.
5 that are >=1.1x,<1.2x faster.
0 that are < 1.1x faster/slower.

Pretty good numbers overall I think.




>
> --
> Joni Orponen
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is Python for Windows compiled with MSVC?

2018-01-31 Thread Ray Donnelly 
On Wed, Jan 31, 2018 at 3:04 PM, Oleg Sivokon  wrote:
> Hello list.
>
> I'll give some background before asking my question in more detail.
>
> I've been tasked with writing some infrastructure code that needs to talk to 
> Kubernetes.  (Kubernetes is a popular software for managing and automating 
> virtualization / containerization of cloud services).  One of the 
> requirements was that the code be written in Python 3.X.
>
> The tasks my code was supposed to perform on Kubernetes would be something 
> like cluster creation from specification, deletion of all or parts of the 
> cluster, providing realtime statistics of cluster usage etc.  There were few 
> prototype scripts written in Bash using kubectl (official client written in 
> Go).
>
> My first reaction was to look for an official client for Kubernetes written 
> in Python. There is one official client for Kubernetes, with a single 
> maintainer, impossible to parse documentation, containing mostly generated 
> code.  It is nigh impossible to use.  Here I need to explain that for 
> whatever reason Kubernetes team decided to write their HTTP API in such a way 
> that the server is "dumb" and the client must be "smart" in order to do 
> anything useful.  For instance, if you have a description of your cluster, 
> you cannot just send this description to the server and hope that it will 
> know how to create the cluster from description.  You need to make multiple 
> API calls (perhaps hundreds of them) to arrange for the server to create the 
> cluster from description.
>
> Since the official client is no help (it really only mirrors the HTTP API), I 
> searched for other clients.  There are two more.  None is in good shape, and 
> none comes even close to being able to do what kubectl can.
>
> There is one more client that shells out calls to kubectl.  It implements 
> only a small subset of kubectl commands, and... it's a lot of parsing of 
> standard output with regular expressions and magic.
>
> Well... I was given a lot of time to investigate other options for dealing 
> with this project, so I decided, what if I can compile kubectl into a shared 
> library and write a Python extension that links against that library.  And, 
> indeed, after few days I came up with such an extension.  It worked!..  On 
> Linux...
>
> Now all I had to do was to re-create my success on Windows (most of the 
> employees in my company use Windows).  At first I thought that I'd 
> cross-compile on Linux using MinGW.  I compiled Go shared library into a DLL, 
> then tried to compile my Python extension and... it didn't work.  I 
> downloaded VirtualBox and some Windows images, etc... tried to compile on 
> Windows.  It didn't work.  I started asking around, and was told that even 
> though for some earlier versions of Python this was kind of possible, for 
> Python 3.5, 3.6 it is not.  You must use MSVC to compile Python extensions.  
> No way around it.
> Now, since Go won't compile with MSVC, I'll have to scrap my project and 
> spend many weeks re-implementing kubectl.
>
> Here's my question: Why?
>
> Why did you choose to use non-free compiler, which also makes 
> cross-compilation impossible?  There wasn't really a reason not to choose 
> MinGW as a way to compile extensions on Windows (Ruby does that, Go uses 
> MinGW, perhaps some others too).  It would've made things like CI and 
> packaging so much easier...  What do Python users / developers get from using 
> MSVC instead?

You can compile extension modules with mingw-w64 just fine (modulus a
few gotchas). The Anaconda Distribution we do this for a few packages,
for example rpy2. You can see the build script used here:
https://github.com/AnacondaRecipes/rpy2-feedstock/blob/master/recipe/bld.bat
(disclaimer: I work for Anaconda Inc on this stuff).

MSYS2 also have mingw-w64 builds of Python that might meet your needs.
It is pretty popular in some parts of the open source on Windows world
(disclaimer: I did a lot of the work for this stuff on MSYS2).

>
> Thank you.
>
> Oleg
> This communication and all information contained in or attached to it is 
> confidential, intended solely for the addressee, may be legally privileged 
> and is the intellectual property of one of the companies of NEX Group plc 
> ("NEX") or third parties. If you are not the intended addressee or receive 
> this message in error, please immediately delete all copies of it and notify 
> the sender. We have taken precautions to minimise the risk of transmitting 
> software viruses, but we advise you to carry out your own virus checks on any 
> attachments. We do not accept liability for any loss or damage caused by 
> software viruses. NEX reserves the right to monitor all communications. We do 
> not accept any legal responsibility for the content of communications, and no 
> communication shall be considered legally binding. Furthermore, if the 
> content of this communication is personal or unconnected with our business, 
> we accept n

Re: [Python-Dev] Python startup time

2018-05-02 Thread Ray Donnelly 
On Wed, May 2, 2018, 4:53 AM Gregory Szorc  wrote:

> On 7/19/2017 12:15 PM, Larry Hastings wrote:
> >
> >
> > On 07/19/2017 05:59 AM, Victor Stinner wrote:
> >> Mercurial startup time is already 45.8x slower than Git whereas tested
> >> Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
> >> developers, with a startup time 2x - 3x slower...
> >
> > When Matt Mackall spoke at the Python Language Summit some years back, I
> > recall that he specifically complained about Python startup time.  He
> > said Python 3 "didn't solve any problems for [them]"--they'd already
> > solved their Unicode hygiene problems--and that Python's slow startup
> > time was already a big problem for them.  Python 3 being /even slower/
> > to start was absolutely one of the reasons why they didn't want to
> upgrade.
> >
> > You might think "what's a few milliseconds matter".  But if you run
> > hundreds of commands in a shell script it adds up.  git's speed is one
> > of the few bright spots in its UX, and hg's comparative slowness here is
> > a palpable disadvantage.
> >
> >
> >> So please continue efforts for make Python startup even faster to beat
> >> all other programming languages, and finally convince Mercurial to
> >> upgrade ;-)
> >
> > I believe Mercurial is, finally, slowly porting to Python 3.
> >
> > https://www.mercurial-scm.org/wiki/Python3
> >
> > Nevertheless, I can't really be annoyed or upset at them moving slowly
> > to adopt Python 3, as Matt's objections were entirely legitimate.
>
> I just now found found this thread when searching the archive for
> threads about startup time. And I was searching for threads about
> startup time because Mercurial's startup time has been getting slower
> over the past few months and this is causing substantial pain.
>
> As I posted back in 2014 [1], CPython's startup overhead was >10% of the
> total CPU time in Mercurial's test suite. And when you factor in the
> time to import modules that get Mercurial to a point where it can run
> commands, it was more like 30%!
>
> Mercurial's full test suite currently runs `hg` ~25,000 times. Using
> Victor's startup time numbers of 6.4ms for 2.7 and 14.5ms for
> 3.7/master, Python startup overhead contributes ~160s on 2.7 and ~360s
> on 3.7/master. Even if you divide this by the number of available CPU
> cores, we're talking dozens of seconds of wall time just waiting for
> CPython to get to a place where Mercurial's first bytecode can execute.
>
> And the problem is worse when you factor in the time it takes to import
> Mercurial's own modules.
>
> As a concrete example, I recently landed a Mercurial patch [2] that
> stubs out zope.interface to prevent the import of 9 modules on every
> `hg` invocation. This "only" saved ~6.94ms for a typical `hg`
> invocation. But this decreased the CPU time required to run the test
> suite on my i7-6700K from ~4450s to ~3980s (~89.5% of original) - a
> reduction of almost 8 minutes of CPU time (and over 1 minute of wall time)!
>
> By the time CPython gets Mercurial to a point where we can run useful
> code, we've already blown most of or past the time budget where humans
> perceive an action/command as instantaneous. If you ignore startup
> overhead, Mercurial's performance compares quite well to Git's for many
> operations. But the reality is that CPython startup overhead makes it
> look like Mercurial is non-instantaneous before Mercurial even has the
> opportunity to execute meaningful code!
>
> Mercurial provides a `chg` program that essentially spins up a daemon
> `hg` process running a "command server" so the `chg` program [written in
> C - no startup overhead] can dispatch commands to an already-running
> Python/`hg` process and avoid paying the startup overhead cost. When you
> run Mercurial's test suite using `chg`, it completes *minutes* faster.
> `chg` exists mainly as a workaround for slow startup overhead.
>
> Changing gears, my day job is maintaining Firefox's build system. We use
> Python heavily in the build system. And again, Python startup overhead
> is problematic. I don't have numbers offhand, but we invoke likely a few
> hundred Python processes as part of building Firefox. It should be
> several thousand. But, we've had to "hack" parts of the build system to
> "batch" certain build actions in single process invocations in order to
> avoid Python startup overhead. This undermines the ability of some build
> tools to formulate a reasonable understanding of the DAG and it causes a
> bit of pain for build system developers and makes it difficult to
> achieve "no-op" and fast incremental builds because we're always
> invoking certain Python processes because we've had to move DAG
> awareness out of the build backend and into Python. At some point, we'll
> likely replace Python code with Rust so the build system is more "pure"
> and easier to maintain and reason about.
>
> I've seen posts in this thread and elsewhere in the CPython development
> universe that c

Re: [Python-Dev] Python startup time

2018-05-03 Thread Ray Donnelly 
On Wed, May 2, 2018 at 6:55 PM, Nathaniel Smith  wrote:
> On Wed, May 2, 2018, 09:51 Gregory Szorc  wrote:
>>
>> Correct me if I'm wrong, but aren't there downsides with regards to C
>> extension compatibility to not having a shared libpython? Or does all the
>> packaging tooling "just work" without a libpython? (It's possible I have
my
>> wires crossed up with something else regarding a statically linked
Python.)
>
>
> IIRC, the rule on Linux is that if you build an extension on a statically
> built python, then it can be imported on a shared python, but not
> vice-versa. Manylinux wheels are therefore always built on a static python
> so that they'll work everywhere. (We should probably clean this up
upstream
> at some point, but there's not a lot of appetite for touching this stuff –
> very obscure, very easy to break things without realizing it, not much
> upside.)
>
> On Windows I don't think there is such a thing as a static build, because
> extensions have to link to the python dll to work at all. And on MacOS I'm
> not sure, though from knowing how their linker works my guess is that all
> extensions act like static extensions do on Linux.

Yes, on Windows there's always a python?.dll.

macOS is an interesting one. For Anaconda 5.0 I read somewhere (how's that
for a useless reference - and perhaps I got the wrong end of the stick)
that Python for all Unixen should use a statically linked interpreter so I
happily went ahead and did that. Of course I tested it against a good few
wheels at the time and everything seemed fine (well, no worse than the
usual binary compatibility woes at least) so I went ahead with it.

Now that Python 3.7 is around the corner we have a chance to re-evaluate
this decision. We have received no binary compat. bugs whatsoever due to
this change (we got a few bugs where people used python-config incorrectly
either directly or via swig or CMake), were we just lucky?

Anyway, it is obviously safer for us to do what upstream does and I will
try to post some benchmarks of static vs shared to the list so we can
discuss it. I guess it is a little late in the release schedule to propose
any such change for 3.7? If not I will try to prepare something. I will
discuss it in depth with the rest of the AD team soon too.

>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
>
https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python linkage on macOS

2018-05-04 Thread Ray Donnelly 
On Fri, May 4, 2018 at 11:00 AM, Antoine Pitrou  wrote:
> On Fri, 04 May 2018 00:21:54 +
> Ray Donnelly  wrote:
>>
>> Yes, on Windows there's always a python?.dll.
>>
>> macOS is an interesting one. For Anaconda 5.0 I read somewhere (how's that
>> for a useless reference - and perhaps I got the wrong end of the stick)
>> that Python for all Unixen should use a statically linked interpreter so I
>> happily went ahead and did that.
>
> A statically linked Python can also be significantly faster (10 to 20%
> IIRC, more perhaps on ARM).  I think you already know about that :-)
>

Indeed, and it worked out well on Intel too. Thanks for the recommendation.

>> Anyway, it is obviously safer for us to do what upstream does and I will
>> try to post some benchmarks of static vs shared to the list so we can
>> discuss it.
>
> I have no idea what our default builds do on macOS, I'll let Ned Deily
> or another mac expert answer (changing the topic in the hope he notices
> this subthread :-)).
>

And thanks for doing this. For the benchmarks I think I should build
Python 3.6.5 (or would 3.7.0b4 be better?) from pyperformance built
each way using the AD scripts and reply here with the results. If I do
not get it done today then I hope to get them ready by Monday.

> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules

2018-07-24 Thread Ray Donnelly 
MSYS2 has two Python ports, msys2 and mingw-w64. I believe Eric was
referring to the mingw-w64 one?

zlib1.dll in C:\Windows\System32 is a packaging error on the part of
whatever put it there and that is what needs to be fixed here. ISVs
need to stop putting anything in that directory as it leads to the
worst form of DLL-hell. You should try to remove that file (back it up
as something may break) and just move on with your work in my opinion.

On Tue, Jul 24, 2018 at 8:15 AM, eryk sun  wrote:
> On Mon, Jul 23, 2018 at 2:31 PM, Eric Le Lay  wrote:
>>
>> I encountered a problem with the Windows packaging of gPodder[1]
>> using msys2:
>
> Are you using regular Windows Python with msys2, or their custom port?
>
> I installed msys2 and used pacman to install Python 3.6. The msys2
> environment names libraries with an "msys-" prefix in the "/usr/bin"
> directory, such as msys-python3.6m.dll, msys-readline7.dll, and
> msys-z.dll (zlib). This is also the application directory of the msys2
> build of Python (i.e. "/usr/bin/python.exe"), so it's the first
> directory in the default DLL search path (ahead of system directories
> and PATH). Unlike Windows Python, msys2 Python does not use the
> alternate search path that replaces the application directory with the
> DLL directory in the search path.
>
> A way to implement this that allows multiple versions of a DLL to be
> loaded in the same process is to use an assembly that includes the DLL
> file in its ".manifest" file. Add the assembly to the
> extension module's #2 manifest (typically embedded, but can be
> ".2"). The system looks for the ""
> subdirectory in the module directory. In Windows 7+ you can also add a
> probing path in a config file (i.e. ".config") [1] that
> extends the SxS search path with up to 9 relative paths, which can be
> up to two levels above the module directory (i.e. "..\..").
>
> [1]: 
> https://docs.microsoft.com/en-us/windows/desktop/SbsCs/application-configuration-files
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Use of objdump within ctypes _get_soname()

2018-10-09 Thread Ray Donnelly 
Hi,

We ran into an issue on the Anaconda Distribution recently where we
added libarchive-c to conda-build (so we can un-compress more source
archive formats than tarfile supports) and everything was good a few
hours, until it hit various CI systems where objdump is not installed.

I was a bit surprised by this dependency and wondered if there'd be
interest in a fallback path that inspects the elf with some fairly
simple python code to determine the soname instead? I have code that
works already - though it could do with and a tidy up - and has been
tested on a variety of architectures. Would CPython be interested in
an attempt to upstream this?

Is it documented anywhere that objdump is needed to load some
extension modules on Linux?


Best regards,

Ray Donnelly,
Anaconda Inc,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of objdump within ctypes _get_soname()

2018-10-10 Thread Ray Donnelly 
I've looked into this more and it is more subtle than my initial
report suggests (as usual).

We have some custom code in AD around find_library() so it looks in
sys.prefix/lib (on Unix) and I had a bug in that code when _get_soname
returned None. I've fixed that now.

I believe there is a condition under which this could go wrong which
is if the soname returned by objdump differs from the filename and
then the filename is not found. This would happen to people with
objdump installed but not to people without it installed, so I feel
the behaviour here should be unified. If you think we should unify it
by implementing a python replacement for this functionality then I'll
get on with a PR (and file that bug report too), the other option is
to just remove this part and have just the fallbacks happen instead?
On Tue, Oct 9, 2018 at 8:13 AM Gregory P. Smith  wrote:
>
>
>
> On Mon, Oct 8, 2018 at 11:59 PM Ray Donnelly  wrote:
>>
>> Hi,
>>
>> We ran into an issue on the Anaconda Distribution recently where we
>> added libarchive-c to conda-build (so we can un-compress more source
>> archive formats than tarfile supports) and everything was good a few
>> hours, until it hit various CI systems where objdump is not installed.
>>
>> I was a bit surprised by this dependency and wondered if there'd be
>> interest in a fallback path that inspects the elf with some fairly
>> simple python code to determine the soname instead? I have code that
>> works already - though it could do with and a tidy up - and has been
>> tested on a variety of architectures. Would CPython be interested in
>> an attempt to upstream this?
>>
>> Is it documented anywhere that objdump is needed to load some
>> extension modules on Linux?
>
>
> Wow, that looks like gross code buried within ctypes.util (which libarchive-c 
> uses) that calls various platforms versions of objdump or equivalent.  File a 
> bugs.python.org issue and submit a PR, native ELF header reading code for 
> this makes sense.
>
> -G
>
>>
>>
>>
>> Best regards,
>>
>> Ray Donnelly,
>> Anaconda Inc,
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python in next Windows 10 update

2019-05-22 Thread Ray Donnelly 
On Thu, May 23, 2019, 12:17 AM Ivan Pozdeev via Python-Dev <
python-dev@python.org> wrote:

> On 22.05.2019 23:52, Steve Dower wrote:
> > On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote:
> >> As someone whose job is to diagnose and fix problems with running
> software:
> >> Are there patches in your release? Do you provide corresponding sources
> and debug symbols for it?
> >
> > You can find the sources at https://github.com/python/cpython :)
> >
> For Anaconda, this is not so, they apply private patches. So I had to make
> sure.
>

There's nothing 'private' about them.
https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe

>
>
> > I'm working on getting debug symbols packaged for the next release, but
> once I do that they'll be exactly the same binaries as in the
> > traditional installer on python.org. (Right now they are two separate
> builds of the same source.)
> >
> > The package on the Store is not a Microsoft build or release of Python -
> it's published by whoever the Windows build manager is at the
> > time. Just to be confusing, it's me right now, but the actual install is
> not owned or managed by Microsoft - just endorsed and linked.
> >
> > Cheers,
> > Steve
>
> --
> Regards,
> Ivan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com