Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
>> On 25 October 2014 23:22, Chris Angelico  wrote:
>>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
 My point is that your "Windows build" would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.

>>>
>>> So you're saying it's impossible to support two compilers?
>>
>> No, rather that Windows currently only has a single supported compiler
>> (excluding cygwin, which is essentially a different OS). Adding a
>> second compiler doesn't just involve adding support for it - which is
>> all that the people offering mingw patches are doing - but also
>> involves going through the whole CPython ecosystem locating the places
>> where there is an implicit assumption that "all Windows builds use the
>> same compiler" and fixing them. I've already pointed out where this is
>> a question for pip and wheel. Whoever wants to add support for a
>> second compiler needs to be willing to do this part of the job as
>> well.
>>
>> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.
> 
> The msvcrt.dlls that MinGW-w64 depends on are those dating back to
> Windows XP SP3 / XP64. Ironically, the official Windows CPython
> doesn't come with any such crt guarantees and you must ensure that the
> same msvcr??.dll is used for *all* extensions. This puts considerable
> strain on extension developers to use the correct (or any) version of
> Visual Studio to build their extensions for CPython on Windows.

We're well aware of this, and it's a big part of why I'm currently migrating 
CPython to build with VC14, which will not have the same binary compatibility 
issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind 
calls into DLLs, so provided the calling conventions match (which they must or 
everything would crash very quickly), it should be relatively easy to build 
compatible extensions with MinGW-w64.

> Also, where are the publicly accessible specifications and other technical
> descriptions that MinGW-w64 would need to implement strong binary
> compatibility with MSVC? As a random example, those for C++ name
> mangling and the PDB file format would be very helpful.

C++ name mangling is always an implementation detail and it changes from 
version to version. Luckily, CPython is entirely in C, so that doesn't matter. 
PDBs are another red herring - you can build a loadable PE file without PDBs.

The full source code for the MSVCRT is available with any version of Visual 
Studio (including the free editions, last time I checked), so feel free to 
check whatever you need to ensure compatibility. I've suggested to the VC team 
that they could get in touch with the MinGW projects and offer to help them 
improve compatibility with MSVC, but unfortunately I don't think anyone will 
take me up on that. I'm happy to research what I can to answer specific 
questions, but there's very little that isn't already publicly available other 
than direct access to the devs.

Cheers,
Steve

>> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Guido van Rossum]

On Saturday, October 25, 2014, Stefan Richthofer 
wrote:

> Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to
> see now that it does not even trigger resurrection, since under
> CPython there are no finalizers executed in ref cycles (i.e. I find my
> objects in gc.garbage).
> So I realize, my xy_cyclic tests are pointless anyway since in cyclic
> gc no resurrection can happen.
>
> > The second problem (with weakref) is different: weakrefs are cleared
> > before __del__ is called, so resurrection doesn't affect the whole
> > process.
> It appears weakrefs are only cleared if this is done by gc (where no
> resurrection can happen anyway). If a resurrection-performing-__del__ is
> just called by ref-count-drop-to-0, weakrefs persist - a behavior that is
> very difficult and inefficient to emulate in Jython, but I'll give it
> some more thoughts...
>
> You shouldn't have to emulate that. The exact behavior of GC is allowed to
vary between systems.


> However thanks for the help!
>
> -Stefan
>
>
> > Gesendet: Sonntag, 26. Oktober 2014 um 01:22 Uhr
> > Von: "Antoine Pitrou" >
> > An: python-dev@python.org 
> > Betreff: Re: [Python-Dev] results of id() and weakref.getweakrefs()
> sometimes break on object resurrection
> >
> >
> > Hello Stefan,
> >
> > On Sun, 26 Oct 2014 00:20:47 +0200
> > "Stefan Richthofer" > wrote:
> > > Hello developers,
> > >
> > > I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
> > > regarding object resurrection.
> >
> > Your runGC() function is buggy, it does not run the GC under CPython.
> > Fix it and the first problem (with id()) disappears.
> >
> > The second problem (with weakref) is different: weakrefs are cleared
> > before __del__ is called, so resurrection doesn't affect the whole
> > process. Add a callback to the weakref and you'll see it is getting
> > called.
> >
> > In other words, CPython behaves as expected. Your concern is
> > appreciated, though.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org 
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (on iPad)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Zachary Ware]

On Sat, Oct 25, 2014 at 6:24 PM, R. David Murray  wrote:
> Note: it can be made even less compelling by making it a lot easier to
> build CPython on Windows without having an MSVC license (which I think
> means not using the GUI, for which I say *yay* :).  I think Zach Ware
> has been working on improving the Windows build process, and I keep
> meaning to give it a try...

So far my improvements have been limited to what it takes to build
after installing prerequisites (and documenting what exactly those
prerequisites are), but on that front things are significantly better
in 3.5, I think.  I will note that it's been possible to build Python
entirely without using the VS GUI (though it still has to be
installed, I think) for quite some time, but hasn't been especially
easy to remember the incantations to do so.  3.5 now has a fairly nice
set of batch scripts (I think; but I (re)wrote them :) that work well
together and are even documented in the PCbuild readme.  I've had
dreams of a set of configure.bat/make.bat scripts (issue16895) to make
things even simpler, but I've put that on hold until after Steve
Dower's major overhaul for VC14 has happened.

One thing I'd like to look into more eventually is seeing what it
would take to build with only the Windows SDK installed (which is free
and has no GUI at all), but I think Steve has mentioned something
about that in connection with the work he's doing -- either way,
things will be different when we switch to VC14 so I'm also putting
that off.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Zachary Ware]

On Sat, Oct 25, 2014 at 7:05 PM, Ray Donnelly  wrote:
> On Sun, Oct 26, 2014 at 12:30 AM, Antoine Pitrou  wrote:
>> On Sat, 25 Oct 2014 19:24:38 -0400
>> "R. David Murray"  wrote:
>>>
>>> I know I for one do not generally test patches on Windows because I
>>> haven't taken the time to learn how to build CPython on it.  Sure, I
>>> could test pure python changes by applying patches to an installed
>>> Python, but that's an ongoing pain and I'd rather learn to build CPython
>>> on Windows and get to use the normal hg tools.
>>>
>>> If I could use a more linux-like toolchain to build CPython on windows,
>>
>> Well, I don't know how "linux-like" you want your toolchain, but FTR
>> you should be able to build from the command line by running
>> "Tools\buildbot\build.bat".
>>
>> I doubt the MinGW case can be much simpler :-)
>>
>
> I beg to differ:
>
> "Tools\buildbot\build.bat" contains:
> @rem Used by the buildbot "compile" step.
> cmd /c Tools\buildbot\external.bat
> call "%VS100COMNTOOLS%vsvars32.bat"
> cmd /c Tools\buildbot\clean.bat
> msbuild PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32
>
> ^ this involves purchasing and installing MS Visual Studio (I'm not
> sure if the Express Edition can be used).

The Express Edition is fine for 32-bit builds.  PCbuild\readme.txt has
full details on which editions are needed for what, and in 3.5 also
has a "quick start guide" (absent from older versions due to a
rewriting of the batch scripts that I did a while back):

1.  Install Microsoft Visual C++ 2010 SP1, any edition.
2.  Install Subversion, and make sure 'svn.exe' is on your PATH.
3.  Install NASM, and make sure 'nasm.exe' is on your PATH.
4.  Run "build.bat -e" to build Python in 32-bit Release configuration.
5.  (Optional, but recommended) Run the test suite with "rt.bat -q".

And really, you can skip step 5 if you don't want to run the tests;
you can skip step 3 if you don't want/need ssl; and you can skip step
2 if you don't want/need bz2, lzma, sqlite3, ssl, or tkinter.
Skipping steps 2 or 3 will cause a lot of angry red text in your build
output, but I hope to solve that problem eventually.

> Without explanations for what each step is doing (I posted those last
> week), on MSYS2:
> Download and run:
> http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download
> From the MSYS2 shell:
> pacman -S git base-devel mingw-w64-x86_64-toolchain 
> mingw-w64-i686-toolchain
> git clone https://github.com/Alexpux/MINGW-packages
> cd MINGW-packages/mingw-w64-python3
> makepkg-mingw -sL --nocheck
> (remove --nocheck to run the testsuite. Also you could put this into a
> batch file if you were so inclined.)
>
> To install the newly built packages, from the MSYS2 shell again:
> pacman -U mingw-w64-*.xz
>
> To run them, you should add /mingw64/bin or /mingw32/bin to your PATH
> (or launch a new shell via mingw32_shell.bat or mingw64_shell.bat)
> Of course, if you don't want to build it from source you can simply issue:
> pacman -S mingw-w64-python3
>
> .. all of the above applies equally to mingw-w64-python2.

I'm failing to see where that's simpler :)

For the record, on the issue at hand, I agree that any effort should
go toward making it possible to build extensions without MSVC.  Once
that problem has been completely solved, then we can consider (long
and hard) building Python itself with something else.  I personally
have not been convinced that there's any good reason to do so, but I'm
not unconvincible :)

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> On Sun, Oct 26, 2014 at 1:45 AM, Steve Dower  
> wrote:
>> Ray Donnelly wrote:
>>> Also, where are the publicly accessible specifications and other technical
>>> descriptions that MinGW-w64 would need to implement strong binary
>>> compatibility with MSVC? As a random example, those for C++ name
>>> mangling and the PDB file format would be very helpful.
>>
>> C++ name mangling is always an implementation detail and it changes from
>> version to version. Luckily, CPython is entirely in C, so that doesn't 
>> matter.
>> PDBs are another red herring - you can build a loadable PE file without PDBs.
>
> Of course C++ can be called from C and that is done in some CPython
> extensions, so it's not a red herring. If we want to talk about strong
> binary compatibility I'd expect the aim would be to intermix freely
> between compilers. We'd like people to be able to debug MinGW-w64 code
> using CDB in Visual Studio if they want to, and on the flipside, to
> have GDB able to read PDB files built by MSVC (actually there's a long
> standing problem when debugging MinGW-w64 code in GDB that stack
> unwinding out of MS built dlls is flaky at best) - so again this is
> not really a red herring. I'm also led to believe that MSVC has a very
> good optimizer so if some project wanted to build certain libraries or
> objects with that for their performance critical paths then I can see
> that as being useful to those projects and their users'.

Binary compatibility that strong is very unlikely to ever happen, and certainly
not with versions of compilers that are being actively developed. It would be 
far
too restrictive to both development teams.

The weaker compatibility of C DLL boundaries is far more achievable - we already
mostly have it, as evidenced by some Python packages working correctly with
mismatched compilers. Soon the CRT will be isolated along the same boundaries,
which is short-term pain for long-term gain.

>>
>> The full source code for the MSVCRT is available with any version of Visual
>> Studio (including the free editions, last time I checked), so feel free to
check
>> whatever you need to ensure compatibility. I've suggested to the VC team that
>> they could get in touch with the MinGW projects and offer to help them 
>> improve
>> compatibility with MSVC, but unfortunately I don't think anyone will take me 
>> up
>> on that. I'm happy to research what I can to answer specific questions, but
>> there's very little that isn't already publicly available other than direct
>> access to the devs.
>
> Under what license? We'd rather have open specifications that
> copyrighted, strictly licensed code that we can't look at for various
> tainting reasons.

As far as I can tell, it's covered by the Visual Studio license, which basically
means you can't redistribute the files (I'm not a lawyer, but I've spent plenty 
of
time talking about licenses to lawyers... not sure how much that counts for :) 
).

Most closed-source Microsoft code is not released under open-source-like 
licenses,
so there's no concept of derivative work, attribution or reciprocation, and 
that's
what appears to cover the CRT sources. "Using" the sources probably counts as 
using
VS, which may trigger some non-commercial clauses if you've got the free version
(but probably not the 30 day trial of the paid version... licenses are weird), 
but
reading them is well within the granted permissions.

The intention of including the sources is to help people with debugging... I 
don't
think it's even possible to rebuild the CRT from them. I do understand the taint
concerns though - until recently, I was operating under rules that made even 
some
documentation "unsafe"...

>> Cheers,
>> Steve
>>
 Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sun, Oct 26, 2014 at 1:45 AM, Steve Dower  wrote:
> Ray Donnelly wrote:
>> On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
>>> On 25 October 2014 23:22, Chris Angelico  wrote:
 On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  
 wrote:
> My point is that your "Windows build" would not have the same behaviour
> as a MSVC-produced Windows build, and so testing it with it would not
> certify that your code would actually be compatible with genuine
> MSVC builds of CPython, which we will not stop supporting.
>

 So you're saying it's impossible to support two compilers?
>>>
>>> No, rather that Windows currently only has a single supported compiler
>>> (excluding cygwin, which is essentially a different OS). Adding a
>>> second compiler doesn't just involve adding support for it - which is
>>> all that the people offering mingw patches are doing - but also
>>> involves going through the whole CPython ecosystem locating the places
>>> where there is an implicit assumption that "all Windows builds use the
>>> same compiler" and fixing them. I've already pointed out where this is
>>> a question for pip and wheel. Whoever wants to add support for a
>>> second compiler needs to be willing to do this part of the job as
>>> well.
>>>
>>> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.
>>
>> The msvcrt.dlls that MinGW-w64 depends on are those dating back to
>> Windows XP SP3 / XP64. Ironically, the official Windows CPython
>> doesn't come with any such crt guarantees and you must ensure that the
>> same msvcr??.dll is used for *all* extensions. This puts considerable
>> strain on extension developers to use the correct (or any) version of
>> Visual Studio to build their extensions for CPython on Windows.
>
> We're well aware of this, and it's a big part of why I'm currently migrating 
> CPython to build with VC14, which will not have the same binary compatibility 
> issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind 
> calls into DLLs, so provided the calling conventions match (which they must 
> or everything would crash very quickly), it should be relatively easy to 
> build compatible extensions with MinGW-w64.

Compatibility going forwards though, right? Still it's great to see
positive steps being made for the future of the Windows platform.

>
>> Also, where are the publicly accessible specifications and other technical
>> descriptions that MinGW-w64 would need to implement strong binary
>> compatibility with MSVC? As a random example, those for C++ name
>> mangling and the PDB file format would be very helpful.
>
> C++ name mangling is always an implementation detail and it changes from 
> version to version. Luckily, CPython is entirely in C, so that doesn't 
> matter. PDBs are another red herring - you can build a loadable PE file 
> without PDBs.

Of course C++ can be called from C and that is done in some CPython
extensions, so it's not a red herring. If we want to talk about strong
binary compatibility I'd expect the aim would be to intermix freely
between compilers. We'd like people to be able to debug MinGW-w64 code
using CDB in Visual Studio if they want to, and on the flipside, to
have GDB able to read PDB files built by MSVC (actually there's a long
standing problem when debugging MinGW-w64 code in GDB that stack
unwinding out of MS built dlls is flaky at best) - so again this is
not really a red herring. I'm also led to believe that MSVC has a very
good optimizer so if some project wanted to build certain libraries or
objects with that for their performance critical paths then I can see
that as being useful to those projects and their users'.

>
> The full source code for the MSVCRT is available with any version of Visual 
> Studio (including the free editions, last time I checked), so feel free to 
> check whatever you need to ensure compatibility. I've suggested to the VC 
> team that they could get in touch with the MinGW projects and offer to help 
> them improve compatibility with MSVC, but unfortunately I don't think anyone 
> will take me up on that. I'm happy to research what I can to answer specific 
> questions, but there's very little that isn't already publicly available 
> other than direct access to the devs.

Under what license? We'd rather have open specifications that
copyrighted, strictly licensed code that we can't look at for various
tainting reasons.

>
> Cheers,
> Steve
>
>>> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Antoine Pitrou]

On Sun, 26 Oct 2014 02:50:39 +0200
"Stefan Richthofer"  wrote:
> Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to
> see now that it does not even trigger resurrection, since under
> CPython there are no finalizers executed in ref cycles (i.e. I find my
> objects in gc.garbage).

Try CPython 3.4 :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Stefan Richthofer]

Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to
see now that it does not even trigger resurrection, since under
CPython there are no finalizers executed in ref cycles (i.e. I find my
objects in gc.garbage).
So I realize, my xy_cyclic tests are pointless anyway since in cyclic
gc no resurrection can happen.

> The second problem (with weakref) is different: weakrefs are cleared
> before __del__ is called, so resurrection doesn't affect the whole
> process.
It appears weakrefs are only cleared if this is done by gc (where no
resurrection can happen anyway). If a resurrection-performing-__del__ is
just called by ref-count-drop-to-0, weakrefs persist - a behavior that is
very difficult and inefficient to emulate in Jython, but I'll give it
some more thoughts...

However thanks for the help!

-Stefan


> Gesendet: Sonntag, 26. Oktober 2014 um 01:22 Uhr
> Von: "Antoine Pitrou" 
> An: python-dev@python.org
> Betreff: Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes 
> break on object resurrection
>
> 
> Hello Stefan,
> 
> On Sun, 26 Oct 2014 00:20:47 +0200
> "Stefan Richthofer"  wrote:
> > Hello developers,
> > 
> > I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
> > regarding object resurrection.
> 
> Your runGC() function is buggy, it does not run the GC under CPython.
> Fix it and the first problem (with id()) disappears.
> 
> The second problem (with weakref) is different: weakrefs are cleared
> before __del__ is called, so resurrection doesn't affect the whole
> process. Add a callback to the weakref and you'll see it is getting
> called.
> 
> In other words, CPython behaves as expected. Your concern is
> appreciated, though.
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
> On 25 October 2014 23:22, Chris Angelico  wrote:
>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
>>> My point is that your "Windows build" would not have the same behaviour
>>> as a MSVC-produced Windows build, and so testing it with it would not
>>> certify that your code would actually be compatible with genuine
>>> MSVC builds of CPython, which we will not stop supporting.
>>>
>>
>> So you're saying it's impossible to support two compilers?
>
> No, rather that Windows currently only has a single supported compiler
> (excluding cygwin, which is essentially a different OS). Adding a
> second compiler doesn't just involve adding support for it - which is
> all that the people offering mingw patches are doing - but also
> involves going through the whole CPython ecosystem locating the places
> where there is an implicit assumption that "all Windows builds use the
> same compiler" and fixing them. I've already pointed out where this is
> a question for pip and wheel. Whoever wants to add support for a
> second compiler needs to be willing to do this part of the job as
> well.
>
> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.

The msvcrt.dlls that MinGW-w64 depends on are those dating back to
Windows XP SP3 / XP64. Ironically, the official Windows CPython
doesn't come with any such crt guarantees and you must ensure that the
same msvcr??.dll is used for *all* extensions. This puts considerable
strain on extension developers to use the correct (or any) version of
Visual Studio to build their extensions for CPython on Windows. Also,
where are the publicly accessible specifications and other technical
descriptions that MinGW-w64 would need to implement strong binary
compatibility with MSVC? As a random example, those for C++ name
mangling and the PDB file format would be very helpful.

>
> Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sun, Oct 26, 2014 at 12:30 AM, Antoine Pitrou  wrote:
> On Sat, 25 Oct 2014 19:24:38 -0400
> "R. David Murray"  wrote:
>>
>> I know I for one do not generally test patches on Windows because I
>> haven't taken the time to learn how to build CPython on it.  Sure, I
>> could test pure python changes by applying patches to an installed
>> Python, but that's an ongoing pain and I'd rather learn to build CPython
>> on Windows and get to use the normal hg tools.
>>
>> If I could use a more linux-like toolchain to build CPython on windows,
>
> Well, I don't know how "linux-like" you want your toolchain, but FTR
> you should be able to build from the command line by running
> "Tools\buildbot\build.bat".
>
> I doubt the MinGW case can be much simpler :-)
>

I beg to differ:

"Tools\buildbot\build.bat" contains:
@rem Used by the buildbot "compile" step.
cmd /c Tools\buildbot\external.bat
call "%VS100COMNTOOLS%vsvars32.bat"
cmd /c Tools\buildbot\clean.bat
msbuild PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32

^ this involves purchasing and installing MS Visual Studio (I'm not
sure if the Express Edition can be used).

Without explanations for what each step is doing (I posted those last
week), on MSYS2:
Download and run:
http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20141003.exe/download
>From the MSYS2 shell:
pacman -S git base-devel mingw-w64-x86_64-toolchain mingw-w64-i686-toolchain
git clone https://github.com/Alexpux/MINGW-packages
cd MINGW-packages/mingw-w64-python3
makepkg-mingw -sL --nocheck
(remove --nocheck to run the testsuite. Also you could put this into a
batch file if you were so inclined.)

To install the newly built packages, from the MSYS2 shell again:
pacman -U mingw-w64-*.xz

To run them, you should add /mingw64/bin or /mingw32/bin to your PATH
(or launch a new shell via mingw32_shell.bat or mingw64_shell.bat)
Of course, if you don't want to build it from source you can simply issue:
pacman -S mingw-w64-python3

.. all of the above applies equally to mingw-w64-python2.

> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Mark Lawrence]

On 26/10/2014 00:24, R. David Murray wrote:

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou  wrote:

On Sun, 26 Oct 2014 09:06:36 +1100
Chris Angelico  wrote:

On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:

How do you know this isn't a problem, since you haven't *tested* with
MSVC?
Why on Earth would you want to test your PEP work with an unsupported
Windows compiler and runtime, rather than with the officially supported
compiler and runtime?


This discussion revolved around supporting MinGW in addition to MSVC.
If it had been supported when I was doing that, I could have spun
myself up a Windows build and tested it.


My point is that your "Windows build" would not have the same behaviour
as a MSVC-produced Windows build, and so testing it with it would not
certify that your code would actually be compatible with genuine
MSVC builds of CPython, which we will not stop supporting.


While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David



MSVC Express Edition 2010 works perfectly for building 3.5 so no license 
needed.  Links to older versions have been pointed out on other threads, 
either here or python-ideas, maybe both?  Or use the command line as 
Antoine pointed out elsewhere.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Guido van Rossum]

On Sat, Oct 25, 2014 at 1:10 PM, Ray Donnelly 
wrote:

> On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower 
> wrote:
> > Building CPython for Windows is not something that needs solving. The
> > culture on Windows is to redistribute binaries, not source, and both the
> > core team and a number of redistributors have this figured out (and it
> will
> > only become easier with VC14 and Python 3.5).
>
> This is the second time you've used the vacuous "culture on Windows"
> argument, now with an added appeal to (vague) authority. That may be
> your opinion and that of some others, but there's a large number of
> people who don't care for using non-Free tools. IMHO building CPython
> on Windows using Open Source toolchains is very much something that
> needs merging upstream and supporting by default. What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.
> If CPython wants to truly call itself an Open Source project then I
> consider being able to compile and cross-compile it with capable Open
> Source toolchains on all major platforms a requirement.
>

Please stop this ridiculous argument. There's no definition of "truly open
source project" that has such a requirement, and if you took it to the
extreme you should not be using Windows at all. I appreciate your concern
that building Python for your favorite platform using your favorite
toolchain doesn't work, and if you have patches (or even bug reports) those
are appreciated. But please take your rhetoric about open source elsewhere.


> > I'd rather see this effort thrown behind compiling extensions, including
> > cross compilation. The ABI is well defined enough that any compiler
> should
> > be usable, especially once the new CRT is in use. However, there is work
> > needed to update the various tool chains to link to VC14's CRT and we
> need
> > to figure out the inconsistencies between tools so we can document and
> work
> > through them.
> >
> > Having different builds of CPython out there will only fragment the
> > community and hurt extension authors far more than it may seem to help.
>

Here's the crux of the matter. We want compiled extension modules
distributed via PyPI to work with the binaries distributed from python.org.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sat, 25 Oct 2014 19:24:38 -0400
"R. David Murray"  wrote:
> 
> I know I for one do not generally test patches on Windows because I
> haven't taken the time to learn how to build CPython on it.  Sure, I
> could test pure python changes by applying patches to an installed
> Python, but that's an ongoing pain and I'd rather learn to build CPython
> on Windows and get to use the normal hg tools.
> 
> If I could use a more linux-like toolchain to build CPython on windows,

Well, I don't know how "linux-like" you want your toolchain, but FTR
you should be able to build from the command line by running
"Tools\buildbot\build.bat".

I doubt the MinGW case can be much simpler :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou  wrote:
> On Sun, 26 Oct 2014 09:06:36 +1100
> Chris Angelico  wrote:
> > On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:
> > > How do you know this isn't a problem, since you haven't *tested* with
> > > MSVC?
> > > Why on Earth would you want to test your PEP work with an unsupported
> > > Windows compiler and runtime, rather than with the officially supported
> > > compiler and runtime?
> > 
> > This discussion revolved around supporting MinGW in addition to MSVC.
> > If it had been supported when I was doing that, I could have spun
> > myself up a Windows build and tested it.
> 
> My point is that your "Windows build" would not have the same behaviour
> as a MSVC-produced Windows build, and so testing it with it would not
> certify that your code would actually be compatible with genuine
> MSVC builds of CPython, which we will not stop supporting.

While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Antoine Pitrou]

Hello Stefan,

On Sun, 26 Oct 2014 00:20:47 +0200
"Stefan Richthofer"  wrote:
> Hello developers,
> 
> I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
> regarding object resurrection.

Your runGC() function is buggy, it does not run the GC under CPython.
Fix it and the first problem (with id()) disappears.

The second problem (with weakref) is different: weakrefs are cleared
before __del__ is called, so resurrection doesn't affect the whole
process. Add a callback to the weakref and you'll see it is getting
called.

In other words, CPython behaves as expected. Your concern is
appreciated, though.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection

[Stefan Richthofer]

Hello developers,

I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3)
regarding object resurrection.
Yes, resurrection is evil, but it is a valid scenario. If an object is
resurrected via its finalizer __del__, sometimes its unique id value as
returned from id() changes. Additionally the list of weak references
pointing to it as returned by weakref.getweakrefs(...) breaks (i.e. is
suddenly empty, assuming it wasn't before).
These issues only arise if the resurrection occurs during
cyclic garbage collection. If the object is finalized because its refcount
drops to zero, everything is fine. See the attached test-file.

Is this behaviour intended or is it a bug? If so, which variant is the bug
and which is right? I can hardly believe that whether id() is preserved
should depend on whether the garbage was cyclic or not.

This might appear of low relevance to you, since no sane program intentionally
performs resurrection. However I originally became aware of the issue
in Jython (where it not only occurs for cyclic garbage but in every
resurrection-case), c.f. http://bugs.jython.org/issue2224.
I am interested in this because I am implementing native gc support
in JyNI and need to understand these details to do it right.


Thanks in advance!

Stefanimport unittest
import gc
import time
import weakref

class ReferentDummy:
def __init__(self, name):
self.name = name

def __str__(self):
return self.name

class ResurrectionDummy:
def __del__(self):
ResurrectionDummy.resurrected = self.toResurrect

class GCDetector():
gcIndex = 0

def __del__(self):
GCDetector.gcIndex += 1

maxGCRun = 10

def runGC():
"""
This is needed for Jython, since theoretically Java gc is not guaranteed to
run if gc.collect is called; the run is only attempted. This method assures
that actually a gc run happens.
"""
currentIndex = GCDetector.gcIndex
gcCount = 0
detector = GCDetector()
detector = None
while currentIndex == GCDetector.gcIndex and gcCount < maxGCRun:
gc.collect()
gcCount += 1
time.sleep(0.1)

class GCTests(unittest.TestCase):
def test_id_after_resurrection(self):
l = ["ab"]
rd = ResurrectionDummy()
rd.toResurrect = l
savedId = id(l)
l = None
rd = None
runGC() #needed for Jython etc, even though no cyclic trash appears
self.assertEqual(id(ResurrectionDummy.resurrected), savedId)

def test_id_after_resurrection_cyclic(self):
#CPython 2.7.5 fails this test
rd = ResurrectionDummy()
l = ["ab", rd]
rd.toResurrect = l
savedId = id(l)
l = None
rd = None
runGC()
self.assertEqual(id(ResurrectionDummy.resurrected), savedId)

def test_weakref_after_resurrection(self):
l = ReferentDummy("ab")
rd = ResurrectionDummy()
rd.toResurrect = l
wref = weakref.ref(l)
self.assertIn(wref, weakref.getweakrefs(l))
l = None
rd = None
runGC() #needed for Jython etc, even though no cyclic trash appears
self.assertIn(wref, weakref.getweakrefs(ResurrectionDummy.resurrected))

def test_weakref_after_resurrection_cyclic(self):
#CPython 2.7.5 fails this test
l = ReferentDummy("ab")
rd = ResurrectionDummy()
rd.toResurrect = l
l.cycleLink = rd
wref = weakref.ref(l)
self.assertIn(wref, weakref.getweakrefs(l))
l = None
rd = None
runGC()
self.assertIn(wref, weakref.getweakrefs(ResurrectionDummy.resurrected))


def test_main():
unittest.main()

if __name__ == "__main__":
test_main()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 25 October 2014 23:22, Chris Angelico  wrote:
> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
>> My point is that your "Windows build" would not have the same behaviour
>> as a MSVC-produced Windows build, and so testing it with it would not
>> certify that your code would actually be compatible with genuine
>> MSVC builds of CPython, which we will not stop supporting.
>>
>
> So you're saying it's impossible to support two compilers?

No, rather that Windows currently only has a single supported compiler
(excluding cygwin, which is essentially a different OS). Adding a
second compiler doesn't just involve adding support for it - which is
all that the people offering mingw patches are doing - but also
involves going through the whole CPython ecosystem locating the places
where there is an implicit assumption that "all Windows builds use the
same compiler" and fixing them. I've already pointed out where this is
a question for pip and wheel. Whoever wants to add support for a
second compiler needs to be willing to do this part of the job as
well.

Handwaving arguments that "it's binary compatible" aren't enough. Prove it.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sun, 26 Oct 2014 09:22:18 +1100
Chris Angelico  wrote:
> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
> > My point is that your "Windows build" would not have the same behaviour
> > as a MSVC-produced Windows build, and so testing it with it would not
> > certify that your code would actually be compatible with genuine
> > MSVC builds of CPython, which we will not stop supporting.
> >
> 
> So you're saying it's impossible to support two compilers?

???

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Ned Deily]

In article ,
 Ned Deily  wrote:

> In article ,
>  David Bolen  wrote:
> > So that's sort of strange.
> Very interesting!  I had been doing some housekeeping on some of my 
> older OS X build systems over the past few days and I've run into the 
> same problem.  In particular, I am seeing this failure on an OS X 10.5.8 
> system (running in a Fusion VM) which I've used for years and from which 
> I have regularly cloned repos from hg.python.org. [...]

Update: after consulting with Donald on IRC, it appears that the problem 
was on the python.org end and is now fixed.  David, is it now working 
again for you?

-- 
 Ned Deily,
 n...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Paul Moore]

On 25 October 2014 21:50, Steve Dower  wrote:
> Ray Donnelly wrote:
>> What is it that you
>> are afraid of if CPython can be compiled out of the box using
>> mingw/MinGW-w64? Why are you fighting so hard against having option.
>
> I'm afraid of users having numpy crash because they're using an MSVC
> CPython instead of a mingw CPython. I'm afraid of users not being able
> to use library A and library B at the same time because A requires MSVC
> CPython and B requires mingw CPython. (I can produce more examples
> if you like, but the general concern is having a fragmented community,
> as I said in my previous post.)

Precisely. Either developers test with *all* supported compilers, or
there is a risk of incompatibilities. Advocates of supporting mingw as
a build system for Python generally do *not* suggest that they are
willing to test for, and deal with, cross-version compatibility
issues. Typically mingw is seen as "another platform" in some sense,
by such advocates, having its own set of supporters and maintainers.

The possibility of extensions built with a mingw-compiled Python
failing when used under a MSVC-built Python is real. It's difficult to
say how big that risk is, but it's certainly there. And I see no-one
offering to be responsible for such compatibility issues (the mingw
supporters generally don't want to set up a MSVC build chain, so it's
difficult to see how they could credibly offer).

> I'm fighting against "having options" because it will suck up the precious
> volunteer time we have and direct it away from where it would be more
> useful, which is making it easier to build extensions with other compilers.

And claiming that it doesn't suck up developer time ignores the point
I made above - *someone* has to deal with any compatibility issues
that come up. As a starter, does the wheel format need to include tags
to distinguish whether the target Python is MSVC-built and
mingw-built? Who will check that? If there is a need, who will work on
the code needed to incorporate that change into wheel, pip, and the
relevant PEPs?

As Steve says, the Python community has a genuine, strong need for
people with mingw expertise working on making it easier to build
*extensions* using mingw, that work with a MSVC-built CPython. If it
were possible to cross-compile compatible extensions on Linux,
projects developed on Linux could supply Windows binaries much more
easily, which would be a huge benefit to the whole Windows Python
community. But the mingw experts don't want to work on that,
preferring to develop patches allowing CPython to be built with mingw.
No objection from me, it's your free time, use it as you wish, but as
a Windows user of Python I can confirm that it's not what I'd like you
to be doing as your contribution to Python.

> I would love to see extensions for Windows built on all platforms. I see no
> value in building Python itself for Windows from different platforms.

Exactly.

> If other core developers agree with you that a more "pure" build of Python is
> worthwhile, then they can go ahead and merge the patches (though I suspect
> most would like to see some traction achieved on a fork first). I think it's
> important that I (as Windows build manager) make my own position clear,
> that's all.

Personally, I'm not a core developer, just a long-time member of this
list and occasional contributor to discussions. But I am also a core
pip developer and a Windows user, and from that perspective I am
acutely aware of the common pain points for Python users on Windows.
And they are all about binary extensions, and none at all about
building Python itself. So in my view, being able to build CPython
using mingw is somewhat interesting from a theoretical perspective,
but of little or no practical value[1] and disruptive in a number of
ways, as mentioned above, to improving the overall experience of
Python users on Windows.

Paul

[1] I note, without trying to make a judgement, that many of the
benefits cited for building with mingw are around "being able to use
free tools" or similar essentially ideological issues.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Terry Reedy]

On 10/25/2014 5:11 PM, Chris Angelico wrote:


It might fragment the community to have multiple different binary
distributions. But it ought to be possible for any person/organization
to say "We're going to make our own build of Python, with these
extension modules, built with this compiler, targeting this platform",
and do everything from source. That might mean they can no longer take
the short-cut of "download someone's MSVC-built extension and use it
as-is", but they should be able to take anyone's extension and build
it on their chosen compiler. Having MinGW as a formally supported
platform would make life a lot easier for people who want to test
CPython patches, for instance - my building and testing of PEP
463-enhanced Python was Linux-only, because I didn't want to try to
set up an entire new buildchain just to try to get a Windows binary
going. There's absolutely no need for that to be binary-compatible
with anything else; as long as it'll run the standard library, it'll
do.


David Murray's unanswered post laid out the path to move in the 
direction you want.  Either take it yourself or try to persuade other 
MinGW fans to do so.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 21:10:23 +0100, Ray Donnelly  
wrote:
> On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower  
> wrote:
> > (Apologies for the short reply, posting from my phone.)
> >
> > "MSVC can continue
> > to be the default compiler used for Python on Windows, none of Roumen's
> > patches change that. They would merely open up the choice for packagers and
> > users to build CPython (and extension modules, thanks to separate patches)
> > with alternate compilers, in cross-compilation or otherwise."
> >
> > Building CPython for Windows is not something that needs solving. The
> > culture on Windows is to redistribute binaries, not source, and both the
> > core team and a number of redistributors have this figured out (and it will
> > only become easier with VC14 and Python 3.5).
> 
> This is the second time you've used the vacuous "culture on Windows"
> argument, now with an added appeal to (vague) authority. That may be
> your opinion and that of some others, but there's a large number of
> people who don't care for using non-Free tools. IMHO building CPython
> on Windows using Open Source toolchains is very much something that
> needs merging upstream and supporting by default. What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.
> If CPython wants to truly call itself an Open Source project then I
> consider being able to compile and cross-compile it with capable Open
> Source toolchains on all major platforms a requirement.

You are doing yourself a disservice by this last statement.  There
really can't be any question that Python is an open source project,
so insinuating that the CPython community is "doing something wrong"
is not going to win you friends and helpers.

A better approach would be to acknowledge that what we are currently
doing works well for supporting Windows (especially since we actually
have some engagement from Microsoft that is *getting some problems
fixed* in ways that help make things more open).

And then say, "wouldn't it be *really cool* if we could also build
CPython using an open source toolchain on Windows out of the box?".  You
might not get instant agreement on that (well, clearly you won't), but
it'd be much more likely you'd start garnering support.

Assume that people are well intentioned, and convince them your
suggestions will make things *even better* using positive arguments.
You might not succeed, but you'll have a much better chance.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Chris Angelico]

On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
> My point is that your "Windows build" would not have the same behaviour
> as a MSVC-produced Windows build, and so testing it with it would not
> certify that your code would actually be compatible with genuine
> MSVC builds of CPython, which we will not stop supporting.
>

So you're saying it's impossible to support two compilers?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sun, 26 Oct 2014 09:06:36 +1100
Chris Angelico  wrote:
> On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:
> > How do you know this isn't a problem, since you haven't *tested* with
> > MSVC?
> > Why on Earth would you want to test your PEP work with an unsupported
> > Windows compiler and runtime, rather than with the officially supported
> > compiler and runtime?
> 
> This discussion revolved around supporting MinGW in addition to MSVC.
> If it had been supported when I was doing that, I could have spun
> myself up a Windows build and tested it.

My point is that your "Windows build" would not have the same behaviour
as a MSVC-produced Windows build, and so testing it with it would not
certify that your code would actually be compatible with genuine
MSVC builds of CPython, which we will not stop supporting.

Therefore, what you and the OP are proposing would not make it
*easier* to ensure cross-platform compatibility but rather *harder*, by
adding another incompatible build configuration to the mix of supported
configurations.

The only remaining question is whether it is worthwhile adding support
for such an additional platform, and given that MinGW is extremely
marginal amongst Windows developers, the answer is IMHO no.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Ned Deily]

In article ,
 David Bolen  wrote:

> David Bolen  writes:
> 
> > which appears to die mid-stream while receiving the manifests.
> >
> > So I'm sort of hoping there might be some record server-side as to why
> > things are falling apart mid-way.
> 
> Just to follow-up to myself, I get the same same error trying to do a
> clone from my own personal XP machine rather than the buildbot (which
> is a VM).  I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2.
> 
> However, the same clones completely successfully under OSX and Linux.
> 
> So that's sort of strange.

Very interesting!  I had been doing some housekeeping on some of my 
older OS X build systems over the past few days and I've run into the 
same problem.  In particular, I am seeing this failure on an OS X 10.5.8 
system (running in a Fusion VM) which I've used for years and from which 
I have regularly cloned repos from hg.python.org.  I spent some time 
yesterday trying to isolate it.  I came to the conclusion that it was 
independent of the version of OpenSSL (identical failures occurred with 
the system's ancient Apple 0.9.7 as well as a newly-build 1.0.1j) and 
independent of the version of hg (at least with two data points, current 
and a year-old version) and seemingly independent of the network 
connection.  I was not able to reproduce the failure on the host OS X 
system (10.10) and I didn't have problems a few days earlier with 
various other OS X releases (10.6.x through 10.9.x) also running in VMs 
on the same host.  I stumbled across a workaround for the problem as I 
was experiencing it:  adding --uncompressed to hg clone eliminated 
failures.  You can get more info on the hg failures by adding 
--traceback and --debugger to the clone command.  After spending way too 
much time on the issue, I was not in the mood to spend more time 
isolating the problem after finding a workaround but if others are also 
seeing it, it might be worth doing.  Sigh.

  $ hg --version
  Mercurial Distributed SCM (version 3.1.2)
  $ hg clone -U http://hg.python.org/cpython cpython
  real URL is https://hg.python.org/cpython
  requesting all changes
  adding changesets
  adding manifests
  transaction abort!
  rollback completed
  abort: connection ended unexpectedly
  $ hg clone --uncompressed -U https://hg.python.org/cpython cpython
  streaming all changes
  10404 files to transfer, 248 MB of data
  transferred 248 MB in 44.4 seconds (5.58 MB/sec)

-- 
 Ned Deily,
 n...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Chris Angelico]

On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:
> How do you know this isn't a problem, since you haven't *tested* with
> MSVC?
> Why on Earth would you want to test your PEP work with an unsupported
> Windows compiler and runtime, rather than with the officially supported
> compiler and runtime?

This discussion revolved around supporting MinGW in addition to MSVC.
If it had been supported when I was doing that, I could have spun
myself up a Windows build and tested it. Since it was (and so far
still is) not, the hassle of hunting down a valid MSVC that could
build for Win XP (as that's what my test box runs) was simply not
worthwhile. My point is that there is no community fragmentation
happening here; the only fragmentation is of binary distribution of
extension modules, and there are several ways in which this needn't be
a problem.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sat, Oct 25, 2014 at 10:52 PM, Antoine Pitrou  wrote:
> On Sat, 25 Oct 2014 21:10:23 +0100
> Ray Donnelly  wrote:
>>
>> This is the second time you've used the vacuous "culture on Windows"
>> argument, now with an added appeal to (vague) authority.
> [...]
>> Why are you fighting so hard against having option.
>> If CPython wants to truly call itself an Open Source project then I
>> consider being able to compile and cross-compile it with capable Open
>> Source toolchains on all major platforms a requirement.
>
> Now *that* sounds vacuous.
>

Maybe you missed where I said "I consider".

> Regarding open source, there's a clear and official definition of it,
> which Python satisfies:
> http://opensource.org/osd-annotated
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sun, 26 Oct 2014 08:53:29 +1100
Chris Angelico  wrote:
> On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou  wrote:
> > And how do you know that it would have worked with MSVC if you only use
> > MinGW?
> > If you want to ensure compatibility with MSVC, you must build with MSVC.
> > There's no working around that.
> 
> Precisely. If you build with MinGW, you can't ensure compatibility
> with MSVC. Reread my post: I gave two examples of situations where
> that isn't a problem.

How do you know this isn't a problem, since you haven't *tested* with
MSVC?
Why on Earth would you want to test your PEP work with an unsupported
Windows compiler and runtime, rather than with the officially supported
compiler and runtime?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sat, 25 Oct 2014 21:10:23 +0100
Ray Donnelly  wrote:
> 
> This is the second time you've used the vacuous "culture on Windows"
> argument, now with an added appeal to (vague) authority.
[...]
> Why are you fighting so hard against having option.
> If CPython wants to truly call itself an Open Source project then I
> consider being able to compile and cross-compile it with capable Open
> Source toolchains on all major platforms a requirement.

Now *that* sounds vacuous.

Regarding open source, there's a clear and official definition of it,
which Python satisfies:
http://opensource.org/osd-annotated

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Chris Angelico]

On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou  wrote:
> And how do you know that it would have worked with MSVC if you only use
> MinGW?
> If you want to ensure compatibility with MSVC, you must build with MSVC.
> There's no working around that.

Precisely. If you build with MinGW, you can't ensure compatibility
with MSVC. Reread my post: I gave two examples of situations where
that isn't a problem.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Antoine Pitrou]

On Sun, 26 Oct 2014 08:11:39 +1100
Chris Angelico  wrote:
> 
> It might fragment the community to have multiple different binary
> distributions. But it ought to be possible for any person/organization
> to say "We're going to make our own build of Python, with these
> extension modules, built with this compiler, targeting this platform",
> and do everything from source. That might mean they can no longer take
> the short-cut of "download someone's MSVC-built extension and use it
> as-is", but they should be able to take anyone's extension and build
> it on their chosen compiler. Having MinGW as a formally supported
> platform would make life a lot easier for people who want to test
> CPython patches, for instance - my building and testing of PEP
> 463-enhanced Python was Linux-only,

And how do you know that it would have worked with MSVC if you only use
MinGW?
If you want to ensure compatibility with MSVC, you must build with MSVC.
There's no working around that.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Chris Angelico]

On Sun, Oct 26, 2014 at 7:50 AM, Steve Dower  wrote:
> Ray Donnelly wrote:
>> What is it that you
>> are afraid of if CPython can be compiled out of the box using
>> mingw/MinGW-w64? Why are you fighting so hard against having option.
>
> I'm afraid of users having numpy crash because they're using an MSVC CPython 
> instead of a mingw CPython. I'm afraid of users not being able to use library 
> A and library B at the same time because A requires MSVC CPython and B 
> requires mingw CPython. (I can produce more examples if you like, but the 
> general concern is having a fragmented community, as I said in my previous 
> post.)
>

It might fragment the community to have multiple different binary
distributions. But it ought to be possible for any person/organization
to say "We're going to make our own build of Python, with these
extension modules, built with this compiler, targeting this platform",
and do everything from source. That might mean they can no longer take
the short-cut of "download someone's MSVC-built extension and use it
as-is", but they should be able to take anyone's extension and build
it on their chosen compiler. Having MinGW as a formally supported
platform would make life a lot easier for people who want to test
CPython patches, for instance - my building and testing of PEP
463-enhanced Python was Linux-only, because I didn't want to try to
set up an entire new buildchain just to try to get a Windows binary
going. There's absolutely no need for that to be binary-compatible
with anything else; as long as it'll run the standard library, it'll
do.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Ray Donnelly]

On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower  wrote:
> (Apologies for the short reply, posting from my phone.)
>
> "MSVC can continue
> to be the default compiler used for Python on Windows, none of Roumen's
> patches change that. They would merely open up the choice for packagers and
> users to build CPython (and extension modules, thanks to separate patches)
> with alternate compilers, in cross-compilation or otherwise."
>
> Building CPython for Windows is not something that needs solving. The
> culture on Windows is to redistribute binaries, not source, and both the
> core team and a number of redistributors have this figured out (and it will
> only become easier with VC14 and Python 3.5).

This is the second time you've used the vacuous "culture on Windows"
argument, now with an added appeal to (vague) authority. That may be
your opinion and that of some others, but there's a large number of
people who don't care for using non-Free tools. IMHO building CPython
on Windows using Open Source toolchains is very much something that
needs merging upstream and supporting by default. What is it that you
are afraid of if CPython can be compiled out of the box using
mingw/MinGW-w64? Why are you fighting so hard against having option.
If CPython wants to truly call itself an Open Source project then I
consider being able to compile and cross-compile it with capable Open
Source toolchains on all major platforms a requirement.

>
> I'd rather see this effort thrown behind compiling extensions, including
> cross compilation. The ABI is well defined enough that any compiler should
> be usable, especially once the new CRT is in use. However, there is work
> needed to update the various tool chains to link to VC14's CRT and we need
> to figure out the inconsistencies between tools so we can document and work
> through them.
>
> Having different builds of CPython out there will only fragment the
> community and hurt extension authors far more than it may seem to help.
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> 
> From: Tony Kelman
> Sent: ‎10/‎25/‎2014 9:06
> To: python-dev@python.org
> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows
>
> I'm several weeks late to this discussion, but I'm glad to see that it
> happened. I'm not a Python developer, and barely a user, but I have several
> years of daily experience compiling complicated scientific software cross-
> platform, particularly with MinGW-w64 for Windows. The Python community,
> both core language and scientific package developers and users, needs to
> act here. The problem is bad and getting worse. Luckily much of the work
> to start solving it has already been done in bits and pieces, it needs
> coordination and participation to come to a conclusion.
>
>> Cross compilation is a valid issue, but I hope that build services like
>> Appveyor also help out here. There is regular talk about the PSF/PyPI
>> providing something similar
>
> AppVeyor is better than nothing (I've been using it since beta), but it's
> a far cry from build services and package management as the Linux world
> knows them. Obtaining and setting up build dependencies quickly and
> repeatably, and finishing the build of a complicated piece of software
> such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
> lies), etc. on a small single-core VM with limited memory and a restrictive
> time limit is often not possible. These problems are solved within Linux
> infrastructure like Koji, Open Build Service, buildd, etc.
>
> MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
> compiling a wide variety of libraries from Linux to Windows, in addition to
> building conventionally on Windows for Windows. The MSYS2 collection of
> MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
> been mentioned. Linux distributions including
> - Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
> - openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
> - Arch https://aur.archlinux.org/packages/?K=mingw
> and others have projects for providing many hundreds of open-source
> packages compiled for Windows. Debian has the cross-compilers available but
> not many packages yet (https://packages.debian.org/search?keywords=mingw).
>
> As a developer of a (compiled) open-source library or application, wouldn't
> you love to be able to build binaries on Linux for Windows? With some work
> and build system patches, you can. For many projects it's a simple matter of
> ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
> only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
> Arch. This is possible with a very, very long set of patches, many of which
> have been submitted by Roumen Petrov to the Python bug tracker - see
> http://bugs.python.org/issue17605 and other issues linked therein. Roumen
> has done a huge amount o

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.

I'm afraid of users having numpy crash because they're using an MSVC CPython 
instead of a mingw CPython. I'm afraid of users not being able to use library A 
and library B at the same time because A requires MSVC CPython and B requires 
mingw CPython. (I can produce more examples if you like, but the general 
concern is having a fragmented community, as I said in my previous post.)

I'm fighting against "having options" because it will suck up the precious 
volunteer time we have and direct it away from where it would be more useful, 
which is making it easier to build extensions with other compilers.

I would love to see extensions for Windows built on all platforms. I see no 
value in building Python itself for Windows from different platforms.

If other core developers agree with you that a more "pure" build of Python is 
worthwhile, then they can go ahead and merge the patches (though I suspect most 
would like to see some traction achieved on a fork first). I think it's 
important that I (as Windows build manager) make my own position clear, that's 
all.

(The rest of your email is purely unsubstantiated opinion, which is okay to 
have, but it doesn't demand any reply so I've omitted it.)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[David Bolen]

As another data point, I've tried cloning randomly selected other
repositories from hg.python.org, and smaller repositories (distutils2,
peps, jython to name a few) are all working fine under XP, even though
with jython for example, the clone takes longer in terms of wall time
than I'll often see cpython fail.(*)

A test of what I presumed was a more comparably sized repository
(features/cdecimal) dies like cpython.

-- David

(*) Overall clone time is probably unrelated anyway since the XP
buildbot traditionally needed 10+min for clones in the past (such as
when the new build script changes were in place and every test used a
clone) and was working fine with that.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[David Bolen]

Donald Stufft  writes:

> I have an idea, can you run https://bpaste.net/show/c5d7cd102f5b and
> tell me what it outputs? Both on a machine that works and one that
> doesn’t.

All but Linux (so XP/7 buildbots, XP standalone, OSX) return:
   ('DHE-RSA-AES128-SHA', 'TLSv1/SSLv3', 128)
My Linux (Ubuntu 12.04) returns:
   ('ECDHE-RSA-AES128-SHA', 'TLSv1/SSLv3', 128)

The script was run under a default Python on each box (2.6 on Windows,
2.7 on OSX and Linux).  I tried 2.6 through 3.1 on my standalone XP with
no change, so I don't think it differs by Python version.  Its not
precisely the same as running hg, since it has its own embedded Python
under Windows, but I installed a source install on my XP box under 2.7
and it fails a clone the same way.

In new news though, I just the same failure on the Win7 buildbot in a
clone test.  In repeated attempts, that's the only one so far.

I also realized that one shared feature is that the XP boxes were using
IPv4 while the other boxes were all IPv6 (an HE tunnel on my side).
Though my earlier Win7 failure was also IPv6.  I manually forced the
Win7 box to use IPv4, but didn't see much difference.  It certainly didn't
start failing like the XP boxes.

Anecdotally, the failing XP attempts appear to be running slower in
general (with lower transfer rates as monitored by my router).  I have
had slow clones work on other boxes, so that's not automatically bad.
But I wonder if it's still some sort of timeout somewhere.

I don't think I currently have an active ssh account, but if there were
a way to test a clone over ssh rather than http perhaps that would be a
useful data point, in terms of eliminating some middlemen processing.

-- David


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 05:45:24 -0700, "Tony Kelman"  wrote:
> As a developer of a (compiled) open-source library or application, wouldn't
> you love to be able to build binaries on Linux for Windows? With some work
> and build system patches, you can. For many projects it's a simple matter of
> ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
> only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
> Arch. This is possible with a very, very long set of patches, many of which
> have been submitted by Roumen Petrov to the Python bug tracker - see
> http://bugs.python.org/issue17605 and other issues linked therein. Roumen
> has done a huge amount of work, and he and others who consider the MinGW-w64
> compiler important will continue to do so. (Thanks a million Roumen!)

Yes, I for one appreciate Roumen's work, even though I'm not currently
in a position to review the patches.

> > I could step in as maintainer for Cygwin and builds based on GCC using
> > mingw* APIs.
> >
> > Regards,
> > Roumen Petrov
> 
> A maintainer has volunteered. Others will help. Can any core developers
> please begin reviewing some of his patches? Even if just to say they need
> to be rebased. The lack of responses on the bug tracker is disheartening
> from an outside perspective. The pile of patches accumulating in external
> MinGW packaging projects is tantamount to a fork of CPython. It won't go
> away since there are dedicated packagers working to keep their MinGW-w64
> builds functional, even in the ad-hoc current state. The patches continue
> piling up, making it more difficult for everyone - except for the core
> Python developers if they continue to ignore the problem. Bring the people
> working on these patches into the fold as contributors. Review the patches.
> It would make Python as a language and a community even more diverse and
> welcoming.

IIUC, our general policy for bringing new platforms into core support,
as opposed to continuing to be a separately maintained "set of patches",
is that there be a "big enough" community of interest (I don't remember
the definition of "big enough") and that there be both committed
maintainers *and* at least one buildbot.

Being able to build windows packages on linux is a compelling argument,
but that only applies to building extensions, not the interpreter.

I would recommend starting with any patches that help MinGW that are not
MinGW specific but instead generally improve the build system and cross
compilation.  There have been a number of such issues opened and
improvements made beyond the MinGW related ones, some coming from
Debian, some coming from the Android community.

So  target those first.  My suggestion would be to pick a patch that is
believed to be commit ready, and come to #python-dev on freenode and
gently bug us until it gets committed.  Then pick the next one, and
repeat.  Working from simplest to more complex would also probably be a
good strategy :)

>From there I'd move on to patches that support using MinGW for building
extensions.  There will probably be useful to also get engaged with the
packaging folks.  And, at this point, we would NEED A BUILDBOT.  That
is, a machine that has whatever tools are required installed such that
tests added to the test suite to test MinGW support in distutils would
run, so we can be sure we don't break anything when making other
changes.

For compiling CPython itself with MinGW, we'd first need to develop a
consensus that we want to support it in core.  I'd say get building
extensions working first, and then make that argument.

By the time a bunch of patches get committed, we should be ready (read:
eager :) to promote someone to do it themselves.  Even if we never
decide to support compiling CPython itself with MinGW, I would hope that
getting it to work for extensions would greatly reduce the number of
patches needed to be maintained outside the tree in order to do so.  And
once at least one MinGW advocate is part of the core team, advocacy
becomes easier :)

--David

PS: one meta comment about the MinGW issues: I scanned a few from the
linked bug, and while the issues split the patches out, which is a great
start, there is no discussion on the issues for the individual patches
to give background and motivation and an explanation of what the patch
is trying to accomplish.  But you probably don't want to waste time on
improving ones that apply *only* to compiling CPython itself unless we
get consensus that we want to support that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

(Apologies for the short reply, posting from my phone.)

"MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise."

Building CPython for Windows is not something that needs solving. The culture 
on Windows is to redistribute binaries, not source, and both the core team and 
a number of redistributors have this figured out (and it will only become 
easier with VC14 and Python 3.5).

I'd rather see this effort thrown behind compiling extensions, including cross 
compilation. The ABI is well defined enough that any compiler should be usable, 
especially once the new CRT is in use. However, there is work needed to update 
the various tool chains to link to VC14's CRT and we need to figure out the 
inconsistencies between tools so we can document and work through them.

Having different builds of CPython out there will only fragment the community 
and hurt extension authors far more than it may seem to help.

Cheers,
Steve

Top-posted from my Windows Phone

From: Tony Kelman
Sent: ‎10/‎25/‎2014 9:06
To: python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.

> Cross compilation is a valid issue, but I hope that build services like
> Appveyor also help out here. There is regular talk about the PSF/PyPI
> providing something similar

AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)

> I could step in as maintainer for Cygwin and builds based on GCC using
> mingw* APIs.
>
> Regards,
> Roumen Petrov

A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ig

Re: [Python-Dev] Status of C compilers for Python on Windows

[Tony Kelman]

I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.


Cross compilation is a valid issue, but I hope that build services like
Appveyor also help out here. There is regular talk about the PSF/PyPI
providing something similar


AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)


I could step in as maintainer for Cygwin and builds based on GCC using
mingw* APIs.

Regards,
Roumen Petrov


A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ignore the problem. Bring the people
working on these patches into the fold as contributors. Review the patches.
It would make Python as a language and a community even more diverse and
welcoming.


Deprecate/remove support for compiling CPython itself with compilers
other than MSVC on Windows


I'm not alone in thinking that this would be a bad idea. MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise.

Sincerely,
Tony

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Donald Stufft]

I have an idea, can you run https://bpaste.net/show/c5d7cd102f5b and
tell me what it outputs? Both on a machine that works and one that
doesn’t.

> On Oct 25, 2014, at 2:14 AM, David Bolen  wrote:
> 
> Donald Stufft  writes:
> 
>> What version of OpenSSL is it using.
> 
> I'm using the pre-built Windows Mercurial installer, but if I unpack
> the included library.zip, the SSLEAY32.DLL shows version 0.9.8r.
> 
> This is from the 3.1.2 install I just did a few hours ago.  It appears
> that hg 2.5.2 on my other XP box also has 0.9.8r.  The prior buildbot
> version (1.6.2) looks like it had 0.9.8o.
> 
> I also got around to trying a manual clone on the Windows 7 buildbot,
> and it worked fine, even with the older hg 1.6.2.
> 
> So it seems to correlate with XP more than anything else at the moment.
> 
> -- David
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Steve Dower]

I was seeing this recently and had to run recover on my repo (not sure what the 
command line is for that - TortoiseHg had a menu). YMMV, but the symptoms sound 
the same.

Cheers,
Steve

Top-posted from my Windows Phone

From: David Bolen
Sent: ‎10/‎24/‎2014 22:01
To: python-dev@python.org
Subject: Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

David Bolen  writes:

> which appears to die mid-stream while receiving the manifests.
>
> So I'm sort of hoping there might be some record server-side as to why
> things are falling apart mid-way.

Just to follow-up to myself, I get the same same error trying to do a
clone from my own personal XP machine rather than the buildbot (which
is a VM).  I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2.

However, the same clones completely successfully under OSX and Linux.

So that's sort of strange.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com