Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis wrote: At first glance, this is a minor issue - passing FILE* pointers across DLL boundaries isn't something I'd normally expect people to do - but look further and you find you're opening a real can of worms. For example, Python has public APIs which take FILE* parameters. So convert them to taking PyWrappedFile * parameters. Easy to say, hard to do. But *that's* the solution for this problem. It's always been like this under Windows and will always be. Changing back to msvcrt so that people must compile their extension with non-standard compilation options it's really *worse* than just requiring msvcrt71 and punt. There's also a free compiler from Microsoft and tons of webpages which say how to compile with it. Or with mingw, even. So, I really believe that the situation is settling down. People are doing what they want to, with some difficulties perhaps, but there's nothing really undoable. If another change has to be pursued, it is to abstract Python from CRT altogether, or at least across boundaries. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
On 2/9/06, Neil Hodgson [EMAIL PROTECTED] wrote: Greg Ewing: But that won't help when you need to deal with third-party code that knows nothing about Python or its wrapped file objects, and calls the CRT (or one of the myriad extant CRTs, chosen at random:-) directly. Can you explain exactly why there is a problem here? Its fairly normal under Windows to build applications that provide a generic plugin interface (think Netscape plugins or COM) that allow the plugins to be built with any compiler and runtime. This has all been thrashed out before, but the issue is passing CRT-allocated objects across DLL boundaries. If you open a file (getting a FILE*) in one DLL, using one CRT, and pass it to a second DLL, linked with a different CRT, the FILE* is not valid in that second CRT, and operations on it will fail. At first glance, this is a minor issue - passing FILE* pointers across DLL boundaries isn't something I'd normally expect people to do - but look further and you find you're opening a real can of worms. For example, Python has public APIs which take FILE* parameters. Further, memory allocation is CRT-managed - allocate memory with one CRT's malloc, and dealloacte it elsewhere, and you have issues. So *any* pointer could be CRT-managed, to some extent. Etc, etc... As a counterexample, however, I've heard reports that you can do a binary edit of the DLLs in the Subversion Python bindings, to change references to python23.dll to python24.dll, and everything still works. Make of that what you will... Also, there are intractable cases, like mod_python. Apache is still built with MSVC6, where Python is built with MSVC7.1. And so, mod_python, as a bridge, has *no* CRT that is officially OK. And yet, it works. I don't know how, maybe the mod_python developers could comment. Anyway, that's the brief summary. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: The postgres example is strange to me as I'd never consider passing a FILE* over a DLL boundary. Maybe this is a Unix/Windows cultural thing due to such practices being more dangerous on Windows. In the specific example, Postgres has a PQprint function that can print a query result to a file; the file was sys.stdout. Also, there is still the redistribution issue: to redistribute msvcr71.dll, you need to own a MSVC license. People that want to use py2exe (or some such) are in trouble: they need to distribute both python25.dll, and msvcr71.dll. They are allowed to distribute the former, but (formally) not allowed to distribute the latter. Link statically. Not sure whether this was a serious suggestion. If pythonxy.dll was statically linked, you would get all the CRT duplication already in extension modules. Given that there are APIs in Python where you have to do malloc/free across the python.dll boundary, you get memory leaks. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: But that won't help when you need to deal with third-party code that knows nothing about Python or its wrapped file objects, and calls the CRT (or one of the myriad extant CRTs, chosen at random:-) directly. Can you explain exactly why there is a problem here? Its fairly normal under Windows to build applications that provide a generic plugin interface (think Netscape plugins or COM) that allow the plugins to be built with any compiler and runtime. COM really solves all problems people might have on Windows. Alas, it is not a cross-platform API. Standard C is cross-platform, so Python uses it in its own APIs. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
On 2/9/06, Martin v. Löwis [EMAIL PROTECTED] wrote: COM really solves all problems people might have on Windows. Taken deliberately out of context, that sounds rather like a claim even Microsoft itself wouldn't make. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis: COM really solves all problems people might have on Windows. COM was partly just a continuation of the practices used for controls, VBXs and other forms of extension. Visual Basic never forced use of a particular compiler or runtime library for extensions so why should Python? It was also easy to debug an extension DLL inside release-mode VB (I can't recall if debug versions of VB were ever readily available) which is something that is more difficult than it should be for Python. Alas, it is not a cross-platform API. Standard C is cross-platform, so Python uses it in its own APIs. The old (pre-XPCOM) Netscape plugin interface was cross-platform and worked with any compiler on Windows. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis: Not sure whether this was a serious suggestion. Yes it is. If pythonxy.dll was statically linked, you would get all the CRT duplication already in extension modules. Given that there are APIs in Python where you have to do malloc/free across the python.dll boundary, you get memory leaks. Memory allocations across DLL boundaries will have to use wrapper functions. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Paul Moore: This has all been thrashed out before, but the issue is passing CRT-allocated objects across DLL boundaries. Yes, that was the first point I addressed through wrapping CRT objects. At first glance, this is a minor issue - passing FILE* pointers across DLL boundaries isn't something I'd normally expect people to do - but look further and you find you're opening a real can of worms. For example, Python has public APIs which take FILE* parameters. So convert them to taking PyWrappedFile * parameters. Further, memory allocation is CRT-managed - allocate memory with one CRT's malloc, and dealloacte it elsewhere, and you have issues. So *any* pointer could be CRT-managed, to some extent. Etc, etc... I thought PyMem_Malloc was the correct call to use for memory allocation now and avoided direct links to the CRT for memory management. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: If pythonxy.dll was statically linked, you would get all the CRT duplication already in extension modules. Given that there are APIs in Python where you have to do malloc/free across the python.dll boundary, you get memory leaks. Memory allocations across DLL boundaries will have to use wrapper functions. Sure, but that is a change to the API. Contributions are welcome, along with a plan how breakage of existing code can be minimized. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: At first glance, this is a minor issue - passing FILE* pointers across DLL boundaries isn't something I'd normally expect people to do - but look further and you find you're opening a real can of worms. For example, Python has public APIs which take FILE* parameters. So convert them to taking PyWrappedFile * parameters. Easy to say, hard to do. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis: Visual Basic never forced use of a particular compiler or runtime library for extensions so why should Python? Do you really not know? Because of API that happens to be defined the way it is. It was rhetorical: Why should Python be inferior to VB? I suppose the answer (hmm, am I allowed to anser my own rhtorical questions?) is that it was originally developed on other operating systems and the Windows port has never been as much of a focus for most contributors. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: I suppose the answer (hmm, am I allowed to anser my own rhtorical questions?) is that it was originally developed on other operating systems and the Windows port has never been as much of a focus for most contributors. That's certainly the case. It is all Mark Hammond's doing still; not much has happened since the original Windows port. The other reason, of course, is that adding *specific* support for Windows will break support of other platforms. Microsoft had no problems breaking support of VB on Linux :-) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis [EMAIL PROTECTED] writes: I just came up with an idea how to resolve the VC versioning problems for good: Python should link with mscvrt.dll (which is part of the operating system), not with the CRT that the compiler provides. To do that, we would need to compile and link with the SDK header files and import libraries, not with the ones that visual studio provides. For that to work, everyone building Python or Python extensions (*) would have to install the Platform SDK (which is available for free, but contains quite a number of bits). Would that be acceptable? Disclaimer: I haven't tried yet whether this would actually work. Regards, Martin (*) For Python extensions, it should be possible to use mingw instead, and configure it for linking against msvcrt. I think think would remove a lot of headaches. Downloading and installing the Platform SDK should not be an issue, imo. The only problem that I see is this: I'm not sure the platform SDK include files (.H and .IDL) are really compatible with VC7.1. I remember that we (on our company, building C++ software) had to 'Unregister the PSDK Directories with Visual Studio' (available from the start menu) before building the stuff, otherwise there were compiler errors. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis [EMAIL PROTECTED] wrote: I just came up with an idea how to resolve the VC versioning problems for good: Python should link with mscvrt.dll (which is part of the operating system), not with the CRT that the compiler provides. Can you elaborate exactly on which versioning problems you think of? For that to work, everyone building Python or Python extensions (*) would have to install the Platform SDK (which is available for free, but contains quite a number of bits). Would that be acceptable? It would complicate the build process and make Python lag behind CRT development (including bugfixes and whatnot) that Microsoft does. You could as well ask to always stick with GCC 2.95 to solve ABI problems, but I don't think it's the correct long time solution. I expect more and more Windows libraries (binary version) to be shipped with dependencies on MSVCR71.DLL. Anyway, it's just a feeling, since I still don't understand which problems you are trying to solve in the first place. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Thomas Heller wrote: I'm not sure the platform SDK include files (.H and .IDL) are really compatible with VC7.1. I remember that we (on our company, building C++ software) had to 'Unregister the PSDK Directories with Visual Studio' (available from the start menu) before building the stuff, otherwise there were compiler errors. This needs some testing, sure. However, I'm fairly confident that Microsoft has fixed/is going to fix whatever issues arise - they want the platform SDK to be usable with a recent compiler (not necessarily the latest one). There was a recent update to the platform SDK (which now comes with both Itanium and AMD64 compilers), so I'm (still) optimistic. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Giovanni Bajo wrote: I just came up with an idea how to resolve the VC versioning problems for good: Python should link with mscvrt.dll (which is part of the operating system), not with the CRT that the compiler provides. Can you elaborate exactly on which versioning problems you think of? I could, but I don't want to elaborate too much. Please google for it - there has been written a lot about it. In short, you cannot really link two different versions of msvcrt (e.g. mscvrt.dll, msvcrt4.dll, msvcr7.dll, msvcr71.dll, mscvrtd.dll, msvcr71d.dll, ...) into a single program, plus you cannot redistribute the CRT unless you are a Visual Studio licensee. This causes problems for extension writers: they need to own the same version of visual studio that Python was built with. It would complicate the build process and make Python lag behind CRT development (including bugfixes and whatnot) that Microsoft does. There isn't really too much development in the CRT, and the little development I can see (e.g. in VS 2005) is rather counter-productive. So ideally, Python should drop usage of the CRT entirely (but getting there will be a long process). Hopefully, P3k will drop usage of stdio for file objects, which will be a big step forward. You could as well ask to always stick with GCC 2.95 to solve ABI problems, but I don't think it's the correct long time solution. I expect more and more Windows libraries (binary version) to be shipped with dependencies on MSVCR71.DLL. Now that VS2005 is out, I doubt that. More and more will also depend on msvcr80.dll. Then, when the next visual studio comes out, you can (probably) add msvcr81.dll to the list of libraries that might be used. This will go on forever, and we cannot win. It's really not using GCC 2.95 which I'm after. It's using /lib/libc.so that I want to. People should be free to use whatever compiler they have access to. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis wrote: To do that, we would need to compile and link with the SDK header files and import libraries, not with the ones that visual studio provides. I withdraw that idea. It appears that the platform SDK doesn't (any longer?) provide an import library for msvrt.dll, and Microsoft documents mscvrt as intended only for system components. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis: So ideally, Python should drop usage of the CRT entirely (but getting there will be a long process). Hopefully, P3k will drop usage of stdio for file objects, which will be a big step forward. You don't need to drop the CRT, just encapsulate it so there is one copy controlled by Python that hands out wrapped objects (file handles, file pointers, memory blocks, others?). These wrappers can only be manipulated through calls back to that owning code that then calls the CRT. Unfortunately this change would itself be incompatible with current extensions. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis wrote: I withdraw that idea. It appears that the platform SDK doesn't (any longer?) provide an import library for msvrt.dll, and Microsoft documents mscvrt as intended only for system components. Insofar as it forms a base on which other separately- compiled pieces of code run, it seems to me that Python itself deserves to be classed as a system component. Although I concede that's probably not quite what Microsoft mean by the term... -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Neil Hodgson wrote: You don't need to drop the CRT, just encapsulate it so there is one copy controlled by Python that hands out wrapped objects (file handles, file pointers, memory blocks, others?). These wrappers can only be manipulated through calls back to that owning code that then calls the CRT. But that won't help when you need to deal with third-party code that knows nothing about Python or its wrapped file objects, and calls the CRT (or one of the myriad extant CRTs, chosen at random:-) directly. I can't see *any* solution to this that works in general. Even if Python itself and all its extensions completely avoid using the CRT, there's still the possibility that two different extensions will use two third-party libraries that were compiled with different CRTs. As far as I can see, Microsoft have created an intractable mess here. Their solution of compile your whole program with the same CRT completely misses the possibility that the whole program may consist of disparate separately- written and separately-compiled parts, and there may be no single person with the ability and/or legal right to compile and link the whole thing. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Greg Ewing wrote: As far as I can see, Microsoft have created an intractable mess here. Their solution of compile your whole program with the same CRT completely misses the possibility that the whole program may consist of disparate separately- written and separately-compiled parts, and there may be no single person with the ability and/or legal right to compile and link the whole thing. Hence, Microsoft's suggesting is entirely different these days: use .NET, and you won't have these versioning problems anymore. I'm getting off-topic... Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Martin v. Löwis: I don't think this would be good enough. I then also need a way to provide extension authors with an API that looks like the CRT, but isn't: they cannot realistically change all their code to use the wrapped objects. In a recent case, somebody tried to passed a FILE* to a postrgres DLL linked with a different CRT; he shouldn't need to change the entire postgres code to use the modified API. The postgres example is strange to me as I'd never consider passing a FILE* over a DLL boundary. Maybe this is a Unix/Windows cultural thing due to such practices being more dangerous on Windows. Also, there is still the redistribution issue: to redistribute msvcr71.dll, you need to own a MSVC license. People that want to use py2exe (or some such) are in trouble: they need to distribute both python25.dll, and msvcr71.dll. They are allowed to distribute the former, but (formally) not allowed to distribute the latter. Link statically. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Linking with mscvrt
Greg Ewing: But that won't help when you need to deal with third-party code that knows nothing about Python or its wrapped file objects, and calls the CRT (or one of the myriad extant CRTs, chosen at random:-) directly. Can you explain exactly why there is a problem here? Its fairly normal under Windows to build applications that provide a generic plugin interface (think Netscape plugins or COM) that allow the plugins to be built with any compiler and runtime. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Linking with mscvrt
I just came up with an idea how to resolve the VC versioning problems for good: Python should link with mscvrt.dll (which is part of the operating system), not with the CRT that the compiler provides. To do that, we would need to compile and link with the SDK header files and import libraries, not with the ones that visual studio provides. For that to work, everyone building Python or Python extensions (*) would have to install the Platform SDK (which is available for free, but contains quite a number of bits). Would that be acceptable? Disclaimer: I haven't tried yet whether this would actually work. Regards, Martin (*) For Python extensions, it should be possible to use mingw instead, and configure it for linking against msvcrt. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
