[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2016-02-20 Thread Mark Mikofski

Mark Mikofski added the comment:

still have the `link.exe` 1561 error without `extra_args=['/DLL']` issue. is 
there a patch? It goes in `distutils._msvccompiler` right?

--
nosy: +bwanamarko

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-14 Thread Steve Dower

Changes by Steve Dower :


--
resolution:  -> out of date
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-03 Thread Christoph Gohlke

Christoph Gohlke added the comment:

Just to confirm: the following script fails with `ImportError: DLL load failed` 
on Python 3.5.0rc2. It fails around the 120th extension. The script passes on 
Python 3.4.3.

Also, Python 3.5.0rc2 (not Python 3.4.3) requires the `extra_postargs=['/DLL']` 
argument to the `link_shared_lib` function, otherwise the linker fails with 
`LINK : fatal error LNK1561: entry point must be defined`.

```
# test_issue24872.py
#
# Build up to 256 C extension modules and import them.
#
# On Python 3.5.0rc2 for Windows this raises "ImportError: DLL load failed:
#   A dynamic link library (DLL) initialization routine failed."
#
# http://bugs.python.org/issue24872

import os
import sys
from importlib import import_module
from distutils import ccompiler

assert sys.platform == 'win32'
assert sys.version_info > (3, 2)

c_source = """/* Minimal Python 3 extension module */
#include "Python.h"
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT, "%s", NULL, -1, NULL, NULL, NULL, NULL, NULL
};
PyMODINIT_FUNC PyInit_%s(void) { return PyModule_Create(); }
"""

cl = ccompiler.new_compiler()

for i in range(256):
name = '_tmp%.3i' % i
try:
os.remove(name+'.pyd')
except FileNotFoundError:
pass
with open(name+'.c', 'w') as fh:
fh.write(c_source % (name, name))
objects = cl.compile([name+'.c'])
cl.link_shared_lib(objects, output_libname=name, extra_postargs=['/DLL'],
   export_symbols=["PyInit_"+name])
for ext in 'c', 'obj', 'exp', 'lib':
os.remove(name+'.'+ext)
os.rename(name+'.dll', name+'.pyd')
import_module(name)
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-02 Thread Christoph Gohlke

Christoph Gohlke added the comment:

> Do you know where that time is being spent?

The incremental linker.

> I'd guess it's probably O(N**2) with the number of obj file

Doesn't seem so. For example the pyrxp 2.1 package contains only 23 C files and 
takes minutes to link. Most packages compile and link reasonably fast.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-02 Thread Steve Dower

Steve Dower added the comment:

It may be possible to dynamically link the Fortran library in a similar 
fashion, though it depends on what code the compiler generates. For Python's 
purposes, statically linking it probably isn't a bad idea.

I'll have to check out whether the FLS limitation still applies, but it seems 
likely. I wasn't aware of it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-02 Thread Christoph Gohlke

Christoph Gohlke added the comment:

I have now recompiled hundreds of Python packages and C/C++/Fortran libraries 
with the `/MT /GL /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib` flags. Many 
packages test OK, only few crashes.

Some more findings:

The /MT flag forces to also statically link the Intel Fortran runtime library.

Some libraries, e.g. OpenCV and Boost DLLs, can not be built with those flags 
(must be /MD).

Although most packages pass unittests (numpy, scipy, matplotlib, etc.), running 
real world scripts, Pandas unit tests, or simple scripts in IPython notebook, 
fail semi-randomly with "ImportError: DLL load failed: A dynamic link library 
(DLL) initialization routine failed". This is reproducible with Python 
3.5.0rc2, 64 and 32 bit, on Windows 10 and 7, 64 bit. It seems to be related to 
the number of extensions and DLLs loaded into the process. This needs more 
testing. Is there still a limit of how many DLLs, which are statically-linked 
to the CRT, can be loaded?




--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-09-02 Thread Steve Dower

Steve Dower added the comment:

The FLS limit is definitely still there, and it still seems to be 127, which 
would explain the issues.

Best fix I have for this is to build with /MD and force vcruntimeXXX.dll to 
always be copied alongside builds. That will solve the installation issue and 
the DLL limit, at the cost of complicating licensing and bundling for packages. 
(The DLL *should* only be loaded once, even if it appears in multiple 
locations.)

I don't think this is necessarily a release blocker though. It may just be 
better for specific packages to do it - e.g. numpy could include whichever 
vcruntime DLL was used to at least satisfy all of its PYDs on any Python 
version. If someone happens to have matched builds of numpy/scipy/pandas (which 
seems fairly likely) and they're all /MD and bundle vcruntime then that's one 
FLS index for a decent number of DLLs.

A "better" solution might involve a pip extension that can detect which VC 
redist version is required when extracting a wheel and download/install it for 
the user (I considered proposing this closer to Python's core, but it really 
needs to be at download/install time and not load time).

Package distros and "packs" are likely to be in a position to distribute the 
dependencies they need and can avoid it. Unfortunately, I don't see a way to do 
that from core without effectively restricting the VC version for most people. 
Hopefully one day I'll get to deal with some easy problems... :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-24 Thread Christoph Gohlke

Christoph Gohlke added the comment:

Two findings regarding the new semi-static linking options:

Distutils now creates libraries (.lib) that may not be readable by subsequent 
versions of Visual C++ 
https://msdn.microsoft.com/en-us/library/0zza0de8.aspx.

Build times and static library sizes significantly increase to a point where it 
becomes impractical in some cases. For example, on my system, the zeromq 
package builds in 90 s on Python 3.4.3, while on Python 3.5rc1 it takes 470 s. 
Building HDF5 static libs with the new options takes 30 minutes (close to 100% 
CPU usage on 8 logical processors) compared to less than 5 minutes for the 
default options.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Marc-Andre Lemburg

Marc-Andre Lemburg added the comment:

If I understand you correctly, the only advantage of using /MT is
not require admin privileges for installation of the VC2015 runtime
libs.

Since VC2015 will be used by a lot of applications in a few months,
and it's likely that MS will ship the runtime as Windows update anyway,
the advantage seems minor.

OTOH, the requirement of linking against external libraries which you
cannot recompile or don't support /MT is rather common and won't
go away. And the need for security updates to the ucrt is rather
inevitable as well based on experience with previous CRTs.

Being able to build a statically linked Python binary is a nice
feature for some special application settings, but given the rather
weak arguments for making this the default, I'm not convinced that
this is a good way forward, esp. not when even MS itself recommends
against doing this.

We can have Python run VCredist during the installation to make
sure the runtime DLLs are available. Then no one will have a problem.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Steve Dower

Steve Dower added the comment:

Unfortunately applocal deployment doesn't solve the versioning issue - you'll 
always need to use VS 2015 to build for Python 3.5.

There are only a few more libraries that are affected by /MT. msvcp140.dll is 
the most likely one here. Doing the same /nodefaultlib dance for that might 
work, but it may not.

All of the shared state like memory allocation and file descriptors are in 
ucrt, which will not change with compiler versions. CPython itself does not 
require any extensions to use VS 2015 as long as they link to ucrtbase and not 
another CRT, but it's looking like the same build process won't work everywhere 
else.

Installing the full runtime requires administrative privileges, and removing 
that requirement was one of the priorities. If certain extensions require it or 
if distros install it that's fine, but I don't want distutils to *help* people 
make wheels that won't work on another machine because the runtime isn't there.

Depending on the size difference, statically linking the C++ parts may not be 
so bad. It's certainly no worse than including it local to Python, and all of 
its potentially shared state should be handled by the ucrt already.

Also, I've already had that OpenSSL issue fixed upstream. You must have an 
older version - I've been building it fine without patching for months now.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Steve Dower

Steve Dower added the comment:

We can't have Python run VCredist because it requires administrative privileges 
and installing Python does not require those. This is actually one of the 
biggest complaints about the current installer, but because it looks like the 
design and not a bug it only comes up when you ask people directly what they'd 
like to change. (The most common bug is pip failing to install/uninstall, which 
is what you see on the issue tracker.)

The other benefits are that PYDs compiled with VS 2016+ (or whatever it's 
called) with these options will work with stock Python 3.5 on a clean machine - 
so if your extension links to python3.dll you can build once for all Python 
versions 3.5 and later and know that it will just copy and run. If you're 
building from source and only have a newer compiler than VC14, you will be able 
to build successfully. Also, we could decide to build some future Python 3.5.x 
with a newer compiler as it should not break any existing extensions (after 
much discussion and testing, obviously, but it is only feasible if we properly 
hide the VC version now).

To achieve these (apart from the last point) with VCredist, we need to be 
installing the current and all future versions of the redist with Python. Hence 
a time machine is required, and I haven't come up with any good way to simulate 
a time machine here.

 not when even MS itself recommends against doing this.

I have separate advice (also from MS, from the same people who have been quoted 
previously, but in private conversations so I can't link to it) that if we want 
any chance of our plugin architecture being VC version independent, this is 
what we have to do. I'm emailing again to get more specific advice, but the 
official guidance has always been biased by wanting people to get the latest 
tools (which is why VC9 disappeared until my team made the case that sometimes 
people can't upgrade). We're still pushing hard to make this an acknowledged 
use case, and don't be surprised if at some point in the future official advice 
says if you allow plugins, do what CPython did to help your users and 
developers.


The /MT == statically linked equivalence is an oversimplification in the 
presence of link-time code generation (/GL and /LTCG options), as we can 
take .obj or .lib files compiled with /MT and still use dynamic linking. The 
difference is we have to do it explicitly, which is what the 
/nodefaultlib:libucrt.lib ucrt.lib options do. If we add concrt140, msvcp140 
and vccorlib140 to that as well (and probably the rest of the versions 
redistributables) then all of them will continue to be dynamically linked.

So basically, all the existing static libs could be built with /MT and still 
have their dependencies dynamically linked if that's what the final linker 
decides to do. In any case, they probably need to be rebuilt with VC14 unless 
the authors have very carefully kept a clean API, in which case it may as well 
be a DLL.

Because we're talking about a plugin architecture here, I think it's actually 
advantageous to use static linking of the C++ runtime. The main difference from 
the C runtime is that the C++ runtime does not have to be shared with the host 
- Python - and the advantage is that state will not be shared with other 
plugins that also happen to use the same version of C++ (or worse, a different 
version with the same name, and now we have a conflict).

I appreciate the problems this causes when trying to link in 3rd-party 
dependencies, but given a choice between making life easier for developers or 
users I have to side with the users (while doing as much as I possibly can to 
make developers lives easier). People installing wheels from Christoph's page 
or PyPI should be able to expect it to work. When pip grows extensions I'll 
certainly look into writing an extension for specifying, detecting and getting 
the required VC redistributable, but I don't want core Python to be burdened 
with shipping the full set of distributables.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Marc-Andre Lemburg

Marc-Andre Lemburg added the comment:

On 17.08.2015 18:24, Steve Dower wrote:
 
 We can't have Python run VCredist because it requires administrative 
 privileges and installing Python does not require those. This is actually one 
 of the biggest complaints about the current installer, but because it looks 
 like the design and not a bug it only comes up when you ask people directly 
 what they'd like to change. (The most common bug is pip failing to 
 install/uninstall, which is what you see on the issue tracker.)

Why not ? People certainly trust the Python installer more than
some script which might want to do this afterwards in order to
get a package working. After all, it's signed by the PSF and
comes with certificate backed verification of authenticity.

 The other benefits are that PYDs compiled with VS 2016+ (or whatever it's 
 called) with these options will work with stock Python 3.5 on a clean machine 
 - so if your extension links to python3.dll you can build once for all Python 
 versions 3.5 and later and know that it will just copy and run. If you're 
 building from source and only have a newer compiler than VC14, you will be 
 able to build successfully. Also, we could decide to build some future Python 
 3.5.x with a newer compiler as it should not break any existing extensions 
 (after much discussion and testing, obviously, but it is only feasible if we 
 properly hide the VC version now).

This would only work for extensions using the stable Python ABI.
The standard approach is to rebuild extension for every minor
release, since the standard Python ABI is not guaranteed to stay
backwards compatible.

 To achieve these (apart from the last point) with VCredist, we need to be 
 installing the current and all future versions of the redist with Python. 
 Hence a time machine is required, and I haven't come up with any good way to 
 simulate a time machine here.

See above. This is really not a very common use case. If you search
for Py_LIMITED_API (which has to be defined to enable the stable
ABI), you'll hardly find any reference to code using it.

 not when even MS itself recommends against doing this.
 
 I have separate advice (also from MS, from the same people who have been 
 quoted previously, but in private conversations so I can't link to it) that 
 if we want any chance of our plugin architecture being VC version 
 independent, this is what we have to do. I'm emailing again to get more 
 specific advice, but the official guidance has always been biased by wanting 
 people to get the latest tools (which is why VC9 disappeared until my team 
 made the case that sometimes people can't upgrade). We're still pushing hard 
 to make this an acknowledged use case, and don't be surprised if at some 
 point in the future official advice says if you allow plugins, do what 
 CPython did to help your users and developers.

Regardless of marketing strategies, the fact that you have to reinstall
Python and all extensions in case there's some bug in the CRT is really
the main argument against doing static linking.

Static linking of the CRT is normally only used in situations where
you don't want to have single file executables without external
dependencies, e.g. for working on arbitrary Windows systems without
having to install anything. It's a valid use case, but not a general
purpose one.

 The /MT == statically linked equivalence is an oversimplification in the 
 presence of link-time code generation (/GL and /LTCG options), as we can 
 take .obj or .lib files compiled with /MT and still use dynamic linking. The 
 difference is we have to do it explicitly, which is what the 
 /nodefaultlib:libucrt.lib ucrt.lib options do. If we add concrt140, 
 msvcp140 and vccorlib140 to that as well (and probably the rest of the 
 versions redistributables) then all of them will continue to be dynamically 
 linked.

 So basically, all the existing static libs could be built with /MT and still 
 have their dependencies dynamically linked if that's what the final linker 
 decides to do. In any case, they probably need to be rebuilt with VC14 unless 
 the authors have very carefully kept a clean API, in which case it may as 
 well be a DLL.

You lost me there. What's the advantage of using /MT when you then add
all of the CRT libs to the set of libs which are dynamically linked ?

Just to clarify:

If I want to ship a C extension compiled for Python 3.5 which links to an
external DLL on the system, I will have to tell the users of the extension
to first run VCredist in order for them to be able to use extension
on their system, since Python 3.5 will not ship with the necessary
CRT DLLs, correct ?

 Because we're talking about a plugin architecture here, I think it's actually 
 advantageous to use static linking of the C++ runtime. The main difference 
 from the C runtime is that the C++ runtime does not have to be shared with 
 the host - Python - and the advantage is that state will not be shared with 
 

[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Steve Dower

Steve Dower added the comment:

 Why not [require administrative privileges]?

Because some people want to use Python and don't have administrative 
permissions on their own machine (e.g. students, employees, researchers, etc.)

 The standard approach is to rebuild extension for every minor
 release, since the standard Python ABI is not guaranteed to stay
 backwards compatible.

The last point I posted lets you rebuild extensions *with a different 
compiler*. So when you upgrade to VC15 you can use that to build your Python 
3.5 extensions instead of VC14, and still ship those extensions to users 
without requiring them to install the VC15 redist. Static linking is part of 
this

 the fact that you have to reinstall
 Python and all extensions in case there's some bug in the CRT is really
 the main argument against doing static linking.

The same argument applies against app-local copies of the runtime, which is why 
I'm trying to avoid these dependencies rather than volunteering to redistribute 
and update them for all extensions.

Also, we already rebuild and release Python due to bugs in OpenSSL, so I see no 
reason why we wouldn't do the same for bugs in the (small) part of the CRT we 
statically link.

 Static linking of the CRT is normally only used in situations where
 you don't want to have single file executables without external
 dependencies, e.g. for working on arbitrary Windows systems without
 having to install anything. It's a valid use case, but not a general
 purpose one.

Normally, yes, but we have a new use case now that the majority of the C 
runtime is preinstalled on machines and versionless.

 What's the advantage of using /MT when you then add
 all of the CRT libs to the set of libs which are dynamically linked ?

The part that is statically linked is only a few functions that are either 
critical to program initialization or specially optimized by the compiler, and 
the rest of the CRT is versionless and managed by the operating system.

To be clear, the DLLs I listed are not required by CPython or most pure-C 
extensions (including, I assume, cffi and Cython). It's only when you start 
using C++ and especially the Microsoft provided libraries for parallelism that 
you need these DLLs.

 If I want to ship a C extension compiled for Python 3.5 which links to an
 external DLL on the system, I will have to tell the users of the extension
 to first run VCredist in order for them to be able to use extension
 on their system, since Python 3.5 will not ship with the necessary
 CRT DLLs, correct ?

Correct, just as it's always been. However, if you want to ship a C extension 
compiled for Python 3.5 which *does not* link to an external DLL, you *do not* 
have to tell your user to install anything.

 includes getting CRT bug fixes by means of OS updates rather than complete
 reinstalls of Python and all your extensions.

Users of Python 3.5 will get CRT updates from the OS. Only a few trivial 
functions are statically linked under this scheme.

 Developers can work around these things, but if we end up requiring them
 to redistribute VCredist with every single package that has external
 dependencies, just to be able to install a binary package,
 because Python itself doesn't ship the necessary DLLs, then something
 is clearly broken for Python on Windows.

 Alternatively, we tell users to install VCredist by hand in case
 they plan to use Python with C extensions. Possible, but not
 very user friendly either.

I proposed above making this an option in the installer. Problem is, it doesn't 
help extensions built with later compilers. Only the extension knows which 
version of the redist is needed.

 With Python 2.7 all this is not an issue, creating yet another road
 block to prevent upgrades :-(

The issue with Python 2.7 is that you need to use VC9. If you think requiring 
everyone to use VC14 for Python 3.5 forever is less of an issue than having to 
recompile static libraries with a new version of the compiler, please say so 
explicitly, as I have not heard anyone claim that and need to hear it said in 
complete seriousness before I have any chance of believing it's true.

 chances are high that this will not actually work out in
 practice. Backwards compatibility is already hard, forward
 compatibility is really really difficult to achieve without
 a time machine.

Agreed. However, the forward compatibility problem we're facing right now seems 
to be external libraries compiled under old compilers - that is, VC10 (to pick 
one example) failed at forward compatibility, and some work is required to fix 
that failure. If we concede defeat now, then we will also fail at forward 
compatibility, and it will take work in the future to correct our failure. I'm 
trying to do as much work now as I can to offset that work for everyone else 
later (and thanks for holding me accountable on this - it's more helpful than 
my apparently frustrated responses probably make it seem :) ).

 BTW: Do you 

[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Christoph Gohlke

Christoph Gohlke added the comment:

The matplotlib extensions compiled with Python 3.5.0rc1 (/MT) are larger than 
those compiled with 3.5.0b4 (/MD). The C++ runtime is statically linked. This 
seems undesirable for the same reasons the UCRT is not linked statically.

In Introducing the Universal CRT [1] James McNellis strongly recommend 
against static linking of the Visual C++ libraries, for both performance and 
serviceability reasons. In Visual Studio 2015 RTM Now Available [2] the same 
author commented that one may deploy the Universal CRT app-locally. Do these 
comments not apply to CPython?

[1] 
http://blogs.msdn.com/b/vcblog/archive/2015/03/03/introducing-the-universal-crt.aspx
[2] 
http://blogs.msdn.com/b/vcblog/archive/2015/07/20/visual-studio-2015-rtm-now-available.aspx

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread Christoph Gohlke

Christoph Gohlke added the comment:

Another /MT only issue: cryptography-1.0 and other libraries depending on 
openssl fail to link to static MT openssl-1.0.1p:

cryptlib.obj : error LNK2001: unresolved external symbol __iob_func

This can be fixed manually [1].

[1] 
http://openssl.6102.n7.nabble.com/Compiling-OpenSSl-Project-with-Visual-Studio-2015-td59416.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



Re: [issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-17 Thread M.-A. Lemburg
On 15.08.2015 22:41, Steve Dower wrote:
 
 Marc-Andre: there are a few concerns with including DLLs that aren't new with 
 any of the 3.5 changes.
 
 * depending on another CRT version is fine *if* it is available (users may 
 have to be told/helped to install the redistributable themselves)
 * CRT state will not be shared between versions. This is most obviously a 
 problem if file descriptors are shared, but may also appear when writing 
 console output, doing floating-point calculations, threading, and memory 
 management.
 * potentially many more issues if C++ is used, but since Python doesn't use 
 C++ this is mainly a concern where you have two DLLs using C++ and different 
 runtimes (the CRT is partially/fully implemented in C++, so issues may 
 theoretically occur with only one DLL using C++, but I'm yet to see any in 
 practice or even identify any specific issues - maybe it's fine? I'm not 
 going to guarantee it myself)

These issues have always existed in the past, but were never a real
problem, AFAIK, since the libraries intended to be used externally
will typically come with e.g. memory management APIs to make sure
they retain ownership of the allocated memory on their heap.

It is quite natural to have to run VCredist as part of an application
installer to make sure that the target system has the right VC runtime
DLLs installed (and the installer will do the checking).

The purpose of having DLLs for the runtime is to reduce overall
size of the components as well as being able to easily address
bugs and security issues in the runtime DLLs *without* having
to recompile and redeploy all components using them.

By forcing or even suggesting statically compiled Python C extensions,
we would break this goal and potentially put our users at risk.

IMO, we should follow the MS recommendations for Deployment in Visual C++
as we did in the past:

https://msdn.microsoft.com/en-us/library/dd293574.aspx


You can statically link a Visual C++ library to an application—that is, compile 
it into the
application—so that you don't have to deploy the Visual C++ library files 
separately. However, we
caution against this approach because statically linked libraries cannot be 
updated in place. If you
use static linking and you want to update a linked library, you have to 
recompile and redeploy your
application.


Perhaps I'm missing something, but if the only advantage of statically
compiling in the runtime is to have users not need to run VCredist
at install time, it's not worth all the added trouble this introduces.

If you are trying to make it possible to compile extensions with
compilers following VC2015, then I also don't think this approach
will work: the new compilers will use a new runtime and so
issues you describe above come into play between the extensions
and the interpreter.

In that scenario, they will create real problems, as far as I
understand, the since the Python C API expects to be able to e.g.
share FDs, internal state such as which locale to assume, or
use and free memory allocated by either the interpreter or the
extension in the resp. other component (e.g. PyArg_ParseTuple()).

So in the end, you'll still have to use the same compiler for
extensions as the one used for compiling CPython to make sure
you don't run into these issues - which is essentially the same
situation as for Python =3.4.

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Christoph Gohlke

Christoph Gohlke added the comment:

FWIW, I rebuilt static libraries for zlib, jbig, jpeg, openjpeg, tiff, webp, 
lcms, and freetype with /MT flag (a tedious task) and was able to build 
matplotlib and Pillow using Python 3.5.0rc1. As expected there is no dependency 
on the vcruntime DLL.

Even if everything is built with /MT and using UCRT, some extensions will still 
depend on version specific Visual C runtime DLLs, e.g. OpenMP, requiring users 
or distributors to install/bundle the VC runtime package.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Steve Dower

Steve Dower added the comment:

I expect that, but most extensions don't seem to be in that category so this 
will help remove the administrator barrier.

Thanks for going through that tedious process. I'll put up a patch later today 
and submit for 3.5.0.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Steve Dower

Steve Dower added the comment:

Actually, on rereading this (during daylight hours, previous response was 
before 6am my time) the patch wouldn't help. I need to write some porting notes 
for rebuilding static libraries with suitable settings. I'll base it on my 
initial post and find somewhere appropriate in the docs (and maybe blog about 
it too for SEO purposes).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Steve Dower

Changes by Steve Dower steve.do...@python.org:


--
components: +Documentation, Extension Modules
priority: release blocker - high

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Christoph Gohlke

Christoph Gohlke added the comment:

Matplotlib and my own extensions are using C++ sources but do not depend on 
msvcp140.dll, just the ucrt. Am I missing something?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com




[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Christoph Gohlke

Christoph Gohlke added the comment:

 Thanks for going through that tedious process
~140 libraries to go.

I hit the wall last night trying to build Boost DLLs. Boost's build tool b2 
does not allow `link=shared runtime-link=static`, hence the `/MT /LTCG 
/NODEFAULTLIB:libucrt.lib ucrt.lib` magic does not work.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Steve Dower

Steve Dower added the comment:

Boost requires C++ anyway doesn't it? So the full redist will be required. 
These options are only useful for pure C sources.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-16 Thread Steve Dower

Steve Dower added the comment:

Probably I'm missing something. Maybe there's a subset of C++ that doesn't rely 
on it - a decent amount of the standard template library is generated at 
compile time.

If the dependency isn't there, it'll be fine.

Do you think it'll be worth having a check box in the installer to get the full 
runtime? Bearing in mind that most won't need it and many won't be able to 
install it? I'm not sure it is, but you've got a better idea of which packages 
are popular and which ones need it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-15 Thread Steve Dower

Changes by Steve Dower steve.do...@python.org:


--
nosy: +larry

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-15 Thread Steve Dower

New submission from Steve Dower:

The C Runtime used for Python 3.5 and later consists of two DLLs. One, 
ucrtbase.dll, is an operating system component that can be assumed to be 
available and up to date (or will be installed by the Python installer).

The second DLL is vcruntimeXXX.dll, where XXX is a version number that is 
directly tied to the version of the compiler that was used to build. 

In order to maintain version-independence from the C Runtime, vcruntime should 
be statically, rather than dynamically, linked. This ensures that extensions 
linking to a different vcruntime will run on machines with builds of Python 
prior to that version being available (i.e. we'd have needed a time machine to 
be able to ensure all future versions of vcruntime are available). This would 
essentially put us back in a place where you need to match compilers to build 
extensions.

To achieve this semi-static linking, the following options must be used.

cl /c /MT /GL file.c
link file.obj /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib

Building with /MT links everything statically, and the last two linker options 
substitute the non-static ucrtbase.dll. /GL and /LTCG (link-time code 
generation) ensures the correct stubs for dynamic linking are created - 
compiling with /MD instead of /MT generates these at compile time instead of 
link time.

One problem appears to be including custom-built static libraries that compiled 
objects with /MD instead of /MT:

cl /C testlib.c /MD
lib testlib.c
cl /C test.c /MT /GL
link test.obj testlib.lib /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib
linker errors occur

These are because testlib.lib pulls in the MSVCRT library (which indirectly 
includes the dynamic vcruntime.lib and ucrt.lib) instead of the LIBCMT library 
(which indirectly includes the static libvcruntime.lib and libucrt.lib). 
Building test.c with /MT pulls in LIBCMT and conflicts ensue.

This may be fixed by using /MD for test.c, but this causes a dependency on 
vcruntimeXXX.dll. It appears that it can also be fixed by excluding MSVCRT from 
the final link step:

link test.obj testlib.lib /LTCG **/NODEFAULTLIB:msvcrt.lib** 
/NODEFAULTLIB:libucrt.lib ucrt.lib

Christoph - I know you've already patched your _msvccompiler.py to use /MD 
instead of /MT, but I wonder if you could try reverting that and adding the 
/NODEFAULTLIB:msvcrt.lib linker option instead and see if that helps? For my 
test cases it's been fine, but you have a much larger collection of libraries 
to work with. I'm very keen to see if this is a satisfactory solution.

FWIW, I've spent a lot of time considering ways we could actually include 
vcruntime*.dll such that old Python versions would include or get future 
versions of the DLL and have come up with nothing. Requiring users to install 
every VC redistributable that is released is unacceptable to me, and I would 
much rather help extension builders to create extensions that do not require 
extra DLLs.

--
assignee: steve.dower
components: Windows
messages: 248646
nosy: cgohlke, paul.moore, steve.dower, tim.golden, zach.ware
priority: release blocker
severity: normal
status: open
title: Add /NODEFAULTLIB:MSVCRT to _msvccompiler
type: behavior
versions: Python 3.5, Python 3.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-15 Thread Christoph Gohlke

Christoph Gohlke added the comment:

Thank you for looking into this.

I just tried '/NODEFAULTLIB:msvcrt.lib' with Pillow and matplotlib but still 
get the linker errors:

tiff.lib(jbig.obj) : error LNK2001: unresolved external symbol __imp_memchr

freetype.lib(ftbase.obj) : error LNK2001: unresolved external symbol 
__imp_strrchr
freetype.lib(truetype.obj) : error LNK2001: unresolved external symbol 
__imp_strstr
freetype.lib(type1.obj) : error LNK2001: unresolved external symbol __imp_memchr
freetype.lib(sfnt.obj) : error LNK2001: unresolved external symbol __imp_memchr

Looks like I'll have to rebuild all the 3rd party libraries with '/MT' and link 
DLLs with '/NODEFAULTLIB:libucrt.lib ucrt.lib' to remove the dependency on 
vcruntimeXXX.dll.

This change should probably be mentioned in the release notes.

Maybe also bring it to the attention of Ilan Schnell (ilan) and Cournapeau 
David (cdavid) who build packages for Anaconda and Canopy.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-15 Thread Marc-Andre Lemburg

Marc-Andre Lemburg added the comment:

What effect does this static linking of the VC runtime have on libraries that 
extension modules link at dynamically ?

E.g. say I have an extension which links against an ODBC driver DLL using a 
different VC runtime than the one used to build Python.

In the past, the has never been an issue, so adding a requirement to rebuild 
external libraries is pretty much a no-go for Python extension writing. You 
often have the case where you cannot rebuild the external libraries you want to 
link against.

--
nosy: +lemburg

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24872] Add /NODEFAULTLIB:MSVCRT to _msvccompiler

2015-08-15 Thread Steve Dower

Steve Dower added the comment:

I guess those imports are expected to come from vcruntime. Rebuilding the 
static libraries may be the only option then, sorry (on the bright side, VC 14 
has much better C99 support than earlier versions, so things should be better 
once we get past the problems caused by change).

Marc-Andre: there are a few concerns with including DLLs that aren't new with 
any of the 3.5 changes.

* depending on another CRT version is fine *if* it is available (users may have 
to be told/helped to install the redistributable themselves)
* CRT state will not be shared between versions. This is most obviously a 
problem if file descriptors are shared, but may also appear when writing 
console output, doing floating-point calculations, threading, and memory 
management.
* potentially many more issues if C++ is used, but since Python doesn't use C++ 
this is mainly a concern where you have two DLLs using C++ and different 
runtimes (the CRT is partially/fully implemented in C++, so issues may 
theoretically occur with only one DLL using C++, but I'm yet to see any in 
practice or even identify any specific issues - maybe it's fine? I'm not going 
to guarantee it myself)

The parts of the new CRT that will be statically linked are (broadly):
* process/DLL initialization
* exception handling
* some functions that are also compiler intrinsics

The first two don't apply for C static libraries, and even seem to be okay if 
the static lib is the one declaring main(). I did make the argument that they 
should be statically linked by default, but the response was that security 
patches may need to be applied to those parts of the code (particularly 
exception handling - corrupting the stack and then raising an exception is a 
great way to jump to whatever code you like). If Python shipped this DLL, it 
would not be updateable anyway, so static linking is a considered risk that 
could cause us to someday ship a security update due to a change in the CRT 
(exactly as has been the case for every other version of Python - nothing new 
here).

The last item is what Christoph just hit - because the compiler may partially 
inline those functions (memchr, strchr, etc.) for fast paths, it needs intimate 
knowledge about how they are implemented. As a result, they go in the 
compiler-version specific DLL and not the generic one. This lets the 
implementations make assumptions about what conditions the compiler has already 
tested and/or proven, but since this could change with compiler version the 
implementation may also change.

Since the functions are stateless, there's no issue if they happen to be 
included in a final DLL multiple times for different versions, but that will 
only happen if the original library is compiled with /MT (I checked - there's 
no way to override the code generated here under /MD other than to change the 
compiler option, but link-time code generation should be able to take 
/MT-compiled libraries and dynamically link them).

So basically, there are no new restrictions on linking to dynamic libraries. 
The biggest concern would be if those libraries depend on msvcr100.dll and 
worked with Python 3.4 (which shipped msvcr100), they may not work on Python 
3.5 unless the user has obtained msvcr100.dll some other way.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24872
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com