[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Thomas Heller thel...@ctypes.org added the comment: I'm not sure this issue is really done. The current state works fine, but there is one problem remaining: There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role. First question: Open a new issue, or discuss it in this one (and repoen it)? -- nosy: +theller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
egaudry e...@fft.be added the comment: Hi Thomas, I think we should open a new issue (child from issue 4120), just to make sure the whole thing remains understandable. Regards, Eloi Thomas Heller wrote: Thomas Heller thel...@ctypes.org added the comment: I'm not sure this issue is really done. The current state works fine, but there is one problem remaining: There are Python extensions that seem to need a manifest pointing to the MSVC runtime libs: dlls that start in-process com servers, like pythoncom.dll (from the PyWin32 extension), and also _ctypes.pyd plays this role. First question: Open a new issue, or discuss it in this one (and repoen it)? -- nosy: +theller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ -- Eloi Gaudry Free Field Technologies Axis Park Louvain-la-Neuve Rue Emile Francqui, 1 B-1435 Mont-Saint Guibert BELGIUM Company Phone: +32 10 487 959 Company Fax: +32 10 454 626 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: I mentioned the problem with pythoncom.dll and suggested a solution in issue7833. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis mar...@v.loewis.de added the comment: First question: Open a new issue, or discuss it in this one (and repoen it)? Please open a new issue. AFAICT, the original issues request (do not embed manifests in pyd files) is now implemented fully. Feel free to leave a link to the new issue here, but please start over describing the problem you are seeing, and in what version you are seeing it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Sridhar Ratnakumar sridh...@activestate.com added the comment: See issue 7556 for the can't use a string pattern on a bytes-like object error. -- nosy: +srid ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Stefan Krah stefan-use...@bytereef.org added the comment: With the latest Python3.1 svn version of msvc9compiler.py, I get a can't use a string pattern on a bytes-like object error. The attached diff fixes the error. -- nosy: +skrah versions: +Python 3.1 -Python 2.6, Python 2.7, Python 3.0 Added file: http://bugs.python.org/file15612/3.1-byte-string.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis mar...@v.loewis.de added the comment: Thanks for the patch. Committed as r76651, r76652, r76653, r76654, r76655 and r76656. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Changes by Martin v. Löwis mar...@v.loewis.de: -- resolution: - accepted status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
egaudry e...@fft.be added the comment: I came across this (very) interesting thread after experiencing some redistribution issue with a python(2.6)-based package built with msvc9 compiler. I used to struggled with the SxS Microsoft policy in the past. After I finally went for the private assembly/crt runtime redistribution solution, I always succeeded to deliver a stand-alone package to any windows host around (I got full control on what's actually build and distribute). Having now (using the disutils module) the manifest file embedded in the python extension actually forbids this redistribution solution, for the reasons that have been posted here (SxS policy), unless a private assembly/crt runtime is provided next to each built extension. IMHO, this is not a convenient (and common) way to redistribute a software. This is why I fully agree with the propositions that were made here, i.e. not embedding manifest into a python extension built with the distutils module and msvc. Could anyone tell us if a decision has been made about such a change ? Thanks, Eloi -- nosy: +egaudry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Marc-Andre Lemburg m...@egenix.com added the comment:
Christoph Gohlke wrote:
Christoph Gohlke cgoh...@uci.edu added the comment:
Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with ImportError: DLL load failed:... if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).
/* Import the testpyd.pyd module. */
#include Python.h
int main(void) {
Py_Initialize();
{
PyObject *p = PyImport_ImportModule(testpyd);
}
Py_Finalize();
return 0;
}
I'm not sure whether this is related to the problem in question.
Does MinGW also embed a manifest in the generated executable ?
If it doesn't, then this is more likely a problem with how
MinGW works per default, than with Python or distutils.
The fact that the above does work without the patch applied
suggests that MinGW does not embed the manifest in the executable
(since the Windows linker then uses the one from the PYD file).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4120
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: My last comment was merely reporting that this patch can break MinGW based build processes. It did surprise some developers that their programs embedding matplotlib, which is now build with the MSVC9 patch, stopped working. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: The MinGW breakage probably comes from the same issue as encountered in http://bugs.python.org/issue4120#msg80908 and #msg80909: the main application, which embeds Matplotlib, does not have a manifest for the MSVCR9 runtimes? That's a problem which should be addressed by adding a manifest in the main application (and manifests can also be separate files called app.exe.manifest if you do not want to embed it). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment:
Apparently the msvc9compiler_stripruntimes_regexp2 patch causes problems
for MinGW users. The following C program is using the Python C API to
import the testpyd extension generated by testpyd.py. When compiled with
MinGW, the program fails with ImportError: DLL load failed:... if the
PYD extension is compiled with MSVC9 and the patch is applied. The
program works if 1) it is compiled with MSVC9, or 2) the testpyd
extension is build without the patch, or 3) the files
Microsoft.VC90.CRT.manifest and msvcr90.dll are placed next to the
executable and the manifest is also embedded into the executable (e.g.
using mt.exe).
/* Import the testpyd.pyd module. */
#include Python.h
int main(void) {
Py_Initialize();
{
PyObject *p = PyImport_ImportModule(testpyd);
}
Py_Finalize();
return 0;
}
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4120
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: There are two levels of testing. First, on a Windows development system with Visual Studio 2008, Python 2.6.2+, and the msvc9compiler_stripruntimes_regexp2.diff patch applied, verify that the embedded manifest in distutil generated PYD extension files is stripped of the WinSxS VC.CRT dependency and that the extensions can still be imported. The attached script (testpyd.py) builds a minimal extension (testpyd.pyd), searches for the WinSxS VC.CRT dependency within the pyd file, and imports the extension. The script should run without exception. Second, verify that distutil generated PYD extensions can also be imported on a test system, which has no Visual Studio 2008 runtime installed in %WINDIR%\WinSxS: Install Windows XP SP3, preferably in a virtual machine. Do not install any additional programs or patches, which might install the Visual Studio 2008 runtime. Then install Python 2.6.2+ using the Install just for me option. Try import the testpyd extension built on the development system: run python.exe -c 'import testpyd' in a directory containing the testpyd.pyd file (not the Python installation directory containing python.exe). This should work without exception. Importing the testpyd extension built without the patch fails on the test system: Traceback (most recent call last): File stdin, line 1, in module ImportError: DLL load failed: This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem. Tested with Python 2.6.3 for Windows 32-bit. -- Added file: http://bugs.python.org/file15161/testpyd.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis mar...@v.loewis.de added the comment: What's a procedure for testing this patch (please be as precise as possible)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Nick Touran n...@partofthething.com added the comment: Just to share my recent experience with this issue: I was attempting to get Python 2.6 working with py4mpi, matplotlib, and pymssql on a 64-bit Windows Vista based HPC cluster via x-copy deployment without the charm of administrative privileges on the server and on my workstation (which is XP). It wasn't easy, but I got it working. I attempted to apply the patch listed here and recompile the packages that complained about Side-by-side configurations, but got in too deep while compiling matplotlib on windows (recall I don't have admin privileges and can't install the prereqs) so I dug deeper. I found this: http://blog.kalmbachnet.de/?postid=80 which suggests removing the publicKeyToken attribute from the manifests to force local DLLs. This would possibly give the same effect as not embedding the manifests in the first place using the patch. So I went in, using VS2008, and removed this attribute from the embedded manifests in python26.dll, python.exe, and the manifest file in C:\python26. I also copied msvcm90.dll, msvcp90.dll, and msvcr90.dll from the 9.0.21022.8 folder in c:\Windows\WinSxS into c:\Python26. With that, matplotlib started working, but pymssql still did not. I then renamed _mssql.pyd to _mssql.dll so that VS2008 could recognize the manifest, removed the publicKeyToken attribute, and renamed it back to _mssql.pyd, but it still complained. Finally, using depends.exe from the internet, I noticed msvcr71.dll was required on my local machine, so I tried copying it over to the remote machine, into the site-packages folder. Bam. It worked. So while I don't really consider this a solution, it definitely worked for what I needed. Any other 3rd party modules that complain in the future in my weird xcopy-deployment will undergo manifest-stripping via VS2008 and other dependency checking via depends.exe. Yay. By the way, the pymssql was compiled with VS2005, which is why it requires msvcr71.dll. If I could rebuild it with VS2008, I imagine that step wouldn't be necessary. While I do have VS2008 on hand, I don't have MS SQL 2020 Super Awesome Developer Edition, or whatever 3GB package installs the required header. -- nosy: +ntouran ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: It may actually be enough to just place the Microsoft.VC90.CRT.manifest file into the Python folder (the one with python.exe and the CRT DLLs). I concur with what Christoph says, that is how the embedded installation works. However, the .pyd files are not in the Python main folder. And therefore they shouldn't have a manifest for the runtimes (because it will look in the current folder relative to the DLL/pyd), because then the runtimes of the main python.exe will be used (which can then be in the same folder as the 'embedded' MSVCR location). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Marc-Andre Lemburg m...@egenix.com added the comment: Looking at the description of manifest files, it appears that just removing the assemblyIdentity-element results in an invalid manifest file: http://msdn.microsoft.com/en-us/library/aa374219(VS.85).aspx It appears that the entire dependency-element referencing the MS VC90 CRT DLL has to be removed in order to make the manifest correct again. Looking at the schema, it's enough to just remove the dependentAssembly-element: http://msdn.microsoft.com/en-us/library/aa375635(VS.85).aspx Reading up on the manifest trouble-shooting page at (near the end of the page): http://msdn.microsoft.com/en-us/library/ms235342.aspx it may actually be enough to just place the Microsoft.VC90.CRT.manifest file into the Python folder (the one with python.exe and the CRT DLLs): If the operating system fails to find the CRT or any other assembly as a shared assembly, it starts looking for the assembly as a private assembly. It searches for private assemblies in the following order: 1. Check the application local folder for a manifest file with name assemblyName.manifest. In this example, the loader tries to find Microsoft.VC90.CRT.manifest in the same folder as appl.exe. If the manifest is found, the loader loads the CRT DLL from the application folder. If the CRT DLL is not found, load fails. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: This patch also removes empty dependentAssembly elements after removing the VC.CRT assemblyIdentity element. It seems not enough to just place the Microsoft.VC90.CRT.manifest and VC runtime DLL files into the Python folder. On a system without the VC runtime installed in winsxs, trying to import a pyd module build without the patch fails with ImportError: DLL load failed: Import works when 1) the manifest and VC runtime DLLs are also placed in the same folder as the pyd file, or 2) the Microsoft Visual C++ 2008 redistributable package is installed, or 3) the pyd file is build with the patch installed. Tested with Windows XP SP3, Python 2.6.3, matplotlib 0.99.1, Visual Studio 2008 SP1. -- Added file: http://bugs.python.org/file15113/msvc9compiler_stripruntimes_regexp2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: Thanks Christoph, those are two important fixes to the patch. I'm +1 on having this in the next 2.6 maintenance release. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Marc-Andre Lemburg m...@egenix.com added the comment: Just a note on the style of the msvc9compiler_stripruntimes_revised.diff patch: * please try to use lines with at most 80 chars only * it would be better to use regexps to do the searchreplace and perhaps add wildcards to catch future SP-versions of VC9 as well Added Distuils as component, since the patch is targeting distutils and extensions, not the Python build process. -- components: +Distutils nosy: +lemburg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: The attached patch uses a regular expression. -- Added file: http://bugs.python.org/file15072/msvc9compiler_stripruntimes_regexp.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Marc-Andre Lemburg m...@egenix.com added the comment: Christoph Gohlke wrote: Christoph Gohlke cgoh...@uci.edu added the comment: The attached patch uses a regular expression. Much better, thanks. -- title: Do not embed manifest files in *.pyd when compiling with MSVC - Do not embed manifest files in *.pyd when compiling withMSVC ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: Could msvc9compiler_stripruntimes.diff still be considered for inclusion in Python 2.6.3 (high priority?)? A lot of .pyd's of third-party extensions are getting dependencies on MSVCR90 added to the .pyd, which makes them not work if the runtimes are not installed into the WinSxS folder and/or when embedding into other applications. For example, PIL encountered this problem recently. In essence, all Windows Python extensions built using setup.py build_ext encounter this problem. What the patch does is edit the manifest file to remove only the dependency on Microsoft.VC90.CRT. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: It probably won't make 2.6.3 as the final is just out. But can we have it for 2.6.4? Here's a link to the relevant discussion on ImageSIG: http:// mail.python.org/pipermail/image-sig/2009-October/005918.html Looking at my local extensions installed, Matplotlib has applied this patch, but PyGame and PIL have embedded manifests with dependencies on the runtime in their .pyd's. The patch now does a text search/replace on an XML file, if you think this is bad style I could rewrit it to use proper XML parsing and writing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christoph Gohlke cgoh...@uci.edu added the comment: There are two easy to fix issues with the msvc9compiler_stripruntimes.diff patch: 1) the dependency for 64-bit VC90.CRT is not removed and 2) the VC90.CRT dependency is removed also from executables, which will fail to run. A revised patch is attached. This is used to build the Matplotlib binaries for Windows and also works for numpy, pygame and PIL. -- nosy: +cgohlke Added file: http://bugs.python.org/file15026/msvc9compiler_stripruntimes_revised.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
johan de taeye jdeta...@users.sourceforge.net added the comment: Is the report posted in: http://groups.google.com/group/comp.lang.python/browse_thread/thread/3ec 6af1279a162ca# also a symptom of the same problem? Johan -- nosy: +jdetaeye ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande k...@tibed.net added the comment: @johan: no it is not a symptom of the same problem. The problem you are having is that the executable which is embedding Python does not have a manifest. If you add a manifest with a dependency on the VC90 runtimes to the main executable, then it should work AFAIK. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
johan de taeye jdeta...@users.sourceforge.net added the comment: @koen: You're aboslutely right. The issue is solved indeed by adding a dependency in the manifest. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Zach Hirsch zhir...@umich.edu added the comment: Without msvc9compiler_stripruntimes.diff, or a patch like it, it doesn't seem possible to use pywin32 (or other binary extensions) with Python-2.6+ installed just for me (i.e., without the VC90 runtime installed globally). Using the python-2.6.1 MSI installed just for me and the pywin32-212 installer, I get the following error when trying to import win32api: Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import win32api Traceback (most recent call last): File stdin, line 1, in module ImportError: DLL load failed: This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem. Removing the manifest from all the .pyd files installed by pywin32 allows me to import win32api without issue. -- nosy: +zhirsch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Christian Heimes [EMAIL PROTECTED] added the comment: Should this patch be applied to 3.0 before the next RC lands? Barry hasn't released RC2 yet. -- nosy: +christian.heimes type: - compile error versions: +Python 2.7, Python 3.0 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: I have now committed Python-2.6-no.manifest.in.pyd.diff as r67120, r67121, and r67122. Thanks for the patch. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: In r67125, r67126, r67127, r67128, r67129, I removed the mt.exe invocation from tcl and tk, and removed the DLLs CRT manifest from the installer. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: I tried to use _CRT_NOFORCE_MANIFEST but i couldn't get it working. There're some infos about this approach at http://blog.m-ri.de/index.php/2008/05/06/hotfix-fuer-usemsprivateassembliesh-und-vc-2008/ , but even with the mentioned __declspec(selectany) int _forceCRTManifest[RTM]; statement, i wasn't able to suppress the manifest dependency :( Microsoft, in its infinite wisdom, decided to compile the CRT itself with those #pragma's enabled. That puts the /manifestdependency linker arguments in msvcrt.lib. When you link against that file you get that CRT manifest entry, whether you like it or not :( Looks like _CRT_NOFORCE_MANIFEST only works if you build your own CRT without the #pragma's. Of course that only applies to the dynamic version of the CRT, when using /MT the manifest hell just vanishes. I'm unsure what the best way to fix this issue for python is... ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande [EMAIL PROTECTED] added the comment: OK, so the define is not going to work. For Python extensions built through distutils, the msvc9compiler_stripruntimes.diff patch will work: it will remove just the dependency on the VC90runtimes. It will leave other assembly dependencies if they are there. For the .pyd's and .dll's in the DLLs folder: I have opened them in a hex editor, and just by hand replaced the assemblyIdentity entry for the VC90runtimes in each file with spaces. I can confirm that this works, also for _tkinter, for which tk85.dll has an additional dependency on Windows Common Controls (though this one was available in my WinSXS). What would be the best way to integrate a search/replace operation like this in the Python build process? Is the manifest generated as a seperate file here first as well, or is it directly embedded? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: What would be the best way to integrate a search/replace operation like this in the Python build process? Is the manifest generated as a seperate file here first as well, or is it directly embedded? What's wrong with Andre Heider's patch? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande [EMAIL PROTECTED] added the comment: Ah, I wasn't thinking it through. It is fine for all .pyd of course, because they don't have any extra dependencies. I was thinking of DLLs with extra dependencies (of which there is one, TK85.dll). There, leaving out the manifest will break it. But indeed, the patch is fine for .pyd building. My mistake! ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Mark Hammond [EMAIL PROTECTED] added the comment: I don't see a problem with this and can see how it would help with private assemblies. -- nosy: +mhammond ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: Skipping the mt.exe call seems fine to me, but there's another solution which might be better: The automatic manifest binding comes from a #pragma comment(linker,/manifestdependency: directive in crtdefs.h. That can be disabled by setting _CRT_NOFORCE_MANIFEST. This way all other manifest dependencies stay in tact and it could be set for *.pyd builds (instead of my patch). TCL's manifest is a different problem. I do not use tkinter, but according to PCbuild/readme.txt it should be possible to get rid of the CRT dependency the same way: just set _CRT_NOFORCE_MANIFEST in the nmake line. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: .. erm, the nmake line in Tools\buildbot\external.bat and Tools/buildbot/external-amd64.bat ;) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande [EMAIL PROTECTED] added the comment: _CRT_NOFORCE_MANIFEST sounds nice, but I'm having some trouble getting it to work for Python extensions. I'm still having manifests generated...? Can you show me how to use it? It is an undocumented option. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande [EMAIL PROTECTED] added the comment: Instead of reverting the patch for Issue 2563, I propose to strip the VC90 runtimes from the manifest (this will preserve other dependencies needed). I checked this by editing tk85.dll and replacing the dependency there with white-space - the Windows Common Controls dependency still worked properly after this. I've attached a patch to strip the runtime from the manifest when building Python extensions (through search/replace). I think somehow the manifests of the .pyd's in DLLs should also be edited, because leaving them out will cause problems if this is done when building tk85.dll. Or perhaps only special care is needed if there ever is a .pyd with additional dependencies? All files with the .pyd extension do not rely on anything else. I don't know enough about the build process to suggest a change for this, except that the build config for the .dll files inside DLLs might also need to be changed. Added file: http://bugs.python.org/file11847/msvc9compiler_stripruntimes.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: For some reason, the linker *does* generate manifest files for all the .pyds when I try it out. Should that worry me? When I delete them, it still can import them just fine. Hm, works for me. I am using MS SDK v6.1 and PCbuild/build.bat. This is not true. For MSVC9SP (VS2008) Microsoft decided that by default, it will still compile against the original CRT You're right, there are defines for that: http://msdn.microsoft.com/en-us/library/cc664727.aspx While that won't load a 2nd copy of the CRT into memory, the embedded manifests of site-packages still have the same problem as the .pyd files now. So if this gets accepted, reverting the patch of Issue 2563 should be considered. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Koen van de Sande [EMAIL PROTECTED] added the comment: The TCL85.dll introduces a subtlety (actually it is TK85.dll). The MSVCRT90 is not the only dependency that can be in the manifest. The Windows.CommonControls.6.0 is used by TK85.dll. So, also for all Python extensions which have additional dependencies beside MSVCRT90, the manifest should still be included. Situation: - .pyd only depends on MSVCRT90: no manifest needed (will fallback to Python 2.6's own manifest and to WinSXS or the installation in the main Python folder) - .pyd has more dependencies: manifest is needed. But, a wild idea, perhaps we could strip MSVCRT90 from the manifest? This is speculation, but perhaps the fallback to the manifest of Python26.exe will still work this way, and the runtimes will only need to be in a single place. Anyone know of an easy-to-build example of a Python extension with extra dependencies? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Changes by Martin v. Löwis [EMAIL PROTECTED]: -- assignee: - loewis priority: - high ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: For some reason, the linker *does* generate manifest files for all the .pyds when I try it out. Should that worry me? When I delete them, it still can import them just fine. FWIW, with the patch alone, the manifest in .DLLs will still be needed, to be able to load tcl85.dll. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: To my surprise there indeed is a vista SP1 box in our test farm. We tested my patch on the following windows machines (each without an installed CRT redist): - xp SP2 and SP3 - xp64 SP2 - server 2003 R2 SP1 - vista with and without SP1 - windows server 2008 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
New submission from Andre Heider [EMAIL PROTECTED]: The MSVC build process currently embeds the .manifest file, which is generated by the MS linker, in the following files: - python.exe - pythonXY.dll - *.pyd The latter is problematic on machines without the MS CRT redistributable installed (No CRT files in %WINDIR%\WinSxS). While this error won't occur when the python package is installed, it will in other cases like in ours: We use the python interpreter within our application and do ship python25.dll. We also ship the MS CRT files and place them next to the main application (MS calls this private assemblies and xcopy deployment). We do not install the redistributable, because it needs admin rights. If a user tries to import socket on this setup it will fail, because the Side by Side / Fusion loader expects the CRT next to the calling Binary (%PYTHONHOME%\DLLs\_socket.pyd) because of the manifest. The solution is to not embed the manifest in the *.pyd Modules. This way the modules just link against msvc*.dll. This works because the windows loader has already mapped the CRT into memory for the process (either through python.exe or pythonXY.dll). This is also the only reliable way i could find to fix the import on machines without redist. Attached you'll find a patch for the MSVC90 build, apply with -p5. It will only patch the release file, PCbuild8/pyd_d.vsprops requires the same fix. Trolltech also uses this approach for its plugins: https://trolltech.com/developer/faqs/faq.2007-10-19.6185511205 -- components: Build files: Python-2.5.2-no.manifest.in.pyd.diff keywords: patch messages: 74723 nosy: aheider severity: normal status: open title: Do not embed manifest files in *.pyd when compiling with MSVC versions: Python 2.5 Added file: http://bugs.python.org/file11783/Python-2.5.2-no.manifest.in.pyd.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: I'm skeptical about this patch. It seems like it could produce an incompatibility with existing installations or deployment procedures. In any case, the VS 2008 project files are not officially supported. If VS 2005 is any similar with VS 2008 wrt. to manifests, I think you can solve your problem by providing a CRT manifest in the directory that has _socket.pyd. I'd be rather interested in seeing the consequences of this approach for Python 2.6. -- nosy: +loewis ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: Yes, i know that v2.5 doesn't officially support MSVC9. But the same problem applies to MSVC8 and its CRT. But in contrast to MSVC9, the CRT8 redist is installed on almost every machine, because alot of software installs it. But thats basically just luck ;) Placing just a .manifest next to *.pyd won't work either. If you're going this route, you have to put the 3 CRT DLLs there too (and thats actually the approach mentioned here: http://msdn.microsoft.com/en-us/library/ms235291.aspx) Then you've got 2 cases: 1) The user does have the redist installed or 2) He does not for 1) The WinSXS installation has a higher priority and the local files are ignored for 2) The files next to the binaries are used (eg c:\python and c:\python\DLLs) Sounds fine, but the latter raises another issue: Now you have 2 copies of the DLLs. Based on these 2 unique file names, windows decides to load 2 copies into memory, one for python.exe|dll and one for *.pyd. As you might see, this gets very nasty because you have 2 different heaps now. Allocating memory from the one instance and freeing it in the other makes the whole process crash. This issue doesn't stop here, here's another situation: Python is compiled with MSVC9. A user has it and MSVC9 SP1 installed, which has a newer CRT version. If he builds site-packages, these files have an embedded manifest with another CRT version than the python interpreter itself. Importing these packages loads a second copy of a CRT into the processes memory, just like described above. I don't know if or for what version this should be addressed for python. I'm just saying that this was a huge issue the last days at the company i work for. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Andre Heider [EMAIL PROTECTED] added the comment: Ok, point taken, so lets aim at v2.6. Should i open a new issue then? Attached you'll find a new diff against v2.6. This time pyd_d.vsprops gets patched too and the linker doesn't even generate a .manifest file. So no, the *.pyd files do not get any manifest info, neither a loose *.pyd.manifest nor an embedded one. I checked a per-user v2.6 install with the CRT manifest pointing at ... I also tried this approach for our software, but i couldn't get it working on w2k3. The solution i am proposing doesn't need a CRT manifest file in ./DLLs. This approach works on w2k3, but i can not test this on vista SP1. Added file: http://bugs.python.org/file11791/Python-2.6-no.manifest.in.pyd.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: I'm just repurposing the issue (despite what I usually claim as a policy) -- resolution: rejected - status: closed - open versions: +Python 2.6 -Python 2.5 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Changes by Martin v. Löwis [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11783/Python-2.5.2-no.manifest.in.pyd.diff ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Martin v. Löwis [EMAIL PROTECTED] added the comment: Yes, i know that v2.5 doesn't officially support MSVC9. But the same problem applies to MSVC8 and its CRT. The point is that even the MSVC8 project files are not supported. They have been included, but they are not actually used for anything. Yes, i know that v2.5 doesn't officially support MSVC9. But the same problem applies to MSVC8 and its CRT. Not necessarily. Take a look at how I deploy Python 2.6. I use a single copy of the DLL, but two copies of the manifest (both referring to the same DLL image). AFAICT, this works fine on XP (but fails on Vista SP1, which complains that the manifest is ill-formed). I'm closing this for 2.5 as rejected; it might cause more problems than it solves, and 2.5.3 will be the last release (i.e. with no release to fix it if it breaks something badly). If you can come up with a working patch for 2.6: that would be more interesting. Would the many manifest files that get generated need to be shipped as well? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4120] Do not embed manifest files in *.pyd when compiling with MSVC
Changes by Martin v. Löwis [EMAIL PROTECTED]: -- resolution: - rejected status: open - closed ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4120 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
