Re: Python loading library containing embedded python...
Martin v. Löwis [EMAIL PROTECTED] wrote: Brennus wrote: I have solved my problem, or at least part of it by downgrading to Visual Studio 2003. For whatever reason, the same steps applied to Visual Studio 2005 result in massive problems. Even the python.exe compiled by VS 2005 throws A/Vs on startup. Does Python have to use Visual Studio? Is this the debug build, by any chance? Microsoft broke support for ISO C in VS2005. The Python trunk has a work-around for that breakage. Good to hear, for my sanity at least. I was actually thinking Microsoft was losing or breaking C on purpose after using VS2005, and even VS2003 to some degree. I think it's some sort of tactic. That's another reason I don't want to continue using VS for Python. I have, of course, seen the MinGW Python patches and tutorials, but I am told I should not use anything but what Python was compiled with. More likely, you managed to mix msvcrt versions somehow. You need to inspect your resulting DLL in depends.exe; please report what it depends on. I can't manage to compile anything with VS when Code Generation is not set to the same type across projects. I guess I will try the VS 2005 route again with this and the VS2005 patches in mind. I am setting Code Generation the same across projects so I figured that meant I was using the same C runtimes. Also notice that you cannot use extension DLLs in your embedded Python: all extension DLLs depend on python2x.dll, so importing one of them leads to multiple copies of the Python code in your interpreter. Are there plans to move away from VS? Well, VS is the the platform compiler, so no. Of course, if you wanted that to happen, you could write a PEP, and explain what alternative compiler (plus alternative build procedure) should be used. I imagine MinGW's build process would be much easier. I *think* I have to keep two Python trees just to *easily* compile static and dynamic import lib files from Python VS. Otherwise, it is quite easy to overwrite a 3 meg static pythoncore.lib with the 200k dynamic/import version. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Brennus wrote: I have a dll/so which embeds python. I can verify it works by compiling it as an executable and adding an appropriate main. Please explain in more detail how you did the embedding. Did you statically link the Python interpreter into your dll/so, or did you use a shared one? How can I make my dll/so with embedded python support use via ctypes? Whether you can do this at all depends on the fine details of embedding: e.g. what operating system, how precisely did you do the embedding, and so on. For example, on many Unix systems, you might end up with two definitions of all the Python symbols, at which point it depends on the precise linker command line options to determine what precisely happens (e.g. which of the definition of the symbol is used in what place). The dll/so must support use in processes which might contain other instances of Python. I can not change that requirement. Running via ctypes is an easy test of this capability (or much of it I suspect). Then your best bet is to *really* embed Python. I.e. link all symbols statically, preferably renaming them (if possible), and make sure that the resulting dll does neither reference nor expose any Python symbols. Given the different linkers on different systems, this is very tricky to implement. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Martin v. Löwis wrote: Brennus wrote: I have a dll/so which embeds python. I can verify it works by compiling it as an executable and adding an appropriate main. Please explain in more detail how you did the embedding. Did you statically link the Python interpreter into your dll/so, or did you use a shared one? I compiled pythoncore.lib. I statically link this to my dll and define Py_BUILD_CORE before including Python.h. The only difference I can see with doing this vs linking the python dll dynamically is that Py_IsInitialized returns true when using the dll. This makes sense, but little else does. How can I make my dll/so with embedded python support use via ctypes? Whether you can do this at all depends on the fine details of embedding: e.g. what operating system, how precisely did you do the embedding, and so on. For example, on many Unix systems, you might end up with two definitions of all the Python symbols, at which point it depends on the precise linker command line options to determine what precisely happens (e.g. which of the definition of the symbol is used in what place). The dll/so must support use in processes which might contain other instances of Python. I can not change that requirement. Running via ctypes is an easy test of this capability (or much of it I suspect). Then your best bet is to *really* embed Python. I.e. link all symbols statically, preferably renaming them (if possible), and make sure that the resulting dll does neither reference nor expose any Python symbols. Given the different linkers on different systems, this is very tricky to implement. For now, I am testing this in Windows with Visual Studio. I will look into renaming the symbols, but I am not sure I understand why this is necessary. Surely two unrelated DLLs can exist in a process with the same function names? If not, I am surprised this problem hasn't been seen or even exploited by plugin competitors more often. Most of the time Py_Initialize and Py_IsInitialized works. High level tests with PyRun_SimpleString() crash in any DLL I use from ctypes, while working fine as an executable. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
I have a dll/so which embeds python. I can verify it works by compiling it as an executable and adding an appropriate main. I tried to write unit tests for this library with ctypes and a simple python script. Access violations and other strange things result. I suspect this is because I am basically embedding python in python at this point. How can I make my dll/so with embedded python support use via ctypes? If Py_NewInterpreter is the answer, why does it hang indefinitely? The dll/so must support use in processes which might contain other instances of Python. I can not change that requirement. Running via ctypes is an easy test of this capability (or much of it I suspect). I have solved my problem, or at least part of it by downgrading to Visual Studio 2003. For whatever reason, the same steps applied to Visual Studio 2005 result in massive problems. Even the python.exe compiled by VS 2005 throws A/Vs on startup. Does Python have to use Visual Studio? I have, of course, seen the MinGW Python patches and tutorials, but I am told I should not use anything but what Python was compiled with. Additionally, since a Python compiled with even VS 2005 is majorly broken, I am wary of using MinGW unless it is officially supported. Are there plans to move away from VS? It's horrible imho. I only upgraded from VS 6 at the recommendation I should use the same compiler used to compile the latest Python. I asked why, citing my own knowledge of struct alignment concerns between compilers, but was vaguely told it was very bad, and that FILE structs can differ. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Brennus [EMAIL PROTECTED] wrote: ... I have solved my problem, or at least part of it by downgrading to Visual Studio 2003. For whatever reason, the same steps applied to Visual Studio 2005 result in massive problems. Even the python.exe compiled by VS 2005 throws A/Vs on startup. I believe the python core developers who do develop on Windows have not even tried VS 2005 and have no intention of so doing for the foreseeable future. Unfortunate -- as a python core developer myself, while I mostly use MacOSX, and secondarily Linux, I'd be happy to lend a hand occasionally with Windows *IF* I could use VS 2005, which I can get for free, rather than any product I'd have to pay for. But, it's not up to me: I use Windows only when I really truly have to (e.g., to check that things work there, too, before publishing articles or books) -- I WON'T commit to spending substantial time and effort on Windows as long as I can use GOOD systems like the Mac or Linux (I used to make a living as a Windows guru, see http://www.aleax.it/TutWin32/ for a trace of that, and exactly because of that I know use Windows as rarely as I can get away with!-). The choice of what Windows environment is best is obviously up to those python core developers who DO use Windows day-in, day-out (such as Tim Peters -- there may be others, but I can't really think of any offhand). Are there plans to move away from VS? It's horrible imho. I only I don't know of any such plans, sadly. You may want to take up the subject on python-dev: few core developers routinely follow this list (or newsgroup, whatever;-). If you volunteer to port the development environment for Windows to X, and can show that the performance with whatever compiler X offers is better than VS's, you may stand a chance. Good luck... you'll need quite a bit;-). Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Brennus wrote: I have solved my problem, or at least part of it by downgrading to Visual Studio 2003. For whatever reason, the same steps applied to Visual Studio 2005 result in massive problems. Even the python.exe compiled by VS 2005 throws A/Vs on startup. Does Python have to use Visual Studio? Is this the debug build, by any chance? Microsoft broke support for ISO C in VS2005. The Python trunk has a work-around for that breakage. I have, of course, seen the MinGW Python patches and tutorials, but I am told I should not use anything but what Python was compiled with. More likely, you managed to mix msvcrt versions somehow. You need to inspect your resulting DLL in depends.exe; please report what it depends on. Also notice that you cannot use extension DLLs in your embedded Python: all extension DLLs depend on python2x.dll, so importing one of them leads to multiple copies of the Python code in your interpreter. Are there plans to move away from VS? Well, VS is the the platform compiler, so no. Of course, if you wanted that to happen, you could write a PEP, and explain what alternative compiler (plus alternative build procedure) should be used. It's horrible imho. I only upgraded from VS 6 at the recommendation I should use the same compiler used to compile the latest Python. I asked why, citing my own knowledge of struct alignment concerns between compilers, but was vaguely told it was very bad, and that FILE structs can differ. The problem really is with multiple copies of msvcrt. The struct alignment in FILE structs is the same across all releases of VC, yet having two copies of msvcrt in memory still might lead to crashes. The problem is that each msvcrt has its own FILE table, so each one has its own stdout, for example. Now, for ages, Microsoft also supports multi-threading in msvcrt, so there was a need to extend struct FILE. They did so by introducing struct FILEX, with an additional lock field. For compatibility, they couldn't change the layout of the existing array of FILE, so for the first 20 FILE values, the lock is stored elsewhere (not in the __iob array, but in a separate _locktable). Now, when a file gets locked, msvcrt checks whether its pointer is between __iob and __iob+20; if it is, it assumes it is its own FILE, and looks for the lock in _locktable, else it assumes it is FILEX, and looks for the lock in the struct itself. If the FILE* really points into __iob of a different msvcrt, that crashes. Similar issues exist for malloc/free: you must not free a pointer allocated by the malloc of a different msvcrt. Each mscvrt has its own heap, and releasing memory to the wrong heap might cause memory leaks. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Brennus wrote: I compiled pythoncore.lib. I statically link this to my dll and define Py_BUILD_CORE before including Python.h. Can you please elaborate? How did you do this? How do you know pythoncore.lib is a static library? What msvcrt is it linked with (use depends.exe to find out)? The only difference I can see with doing this vs linking the python dll dynamically is that Py_IsInitialized returns true when using the dll. This makes sense, but little else does. You should explicitly call Py_Initialize in your embedded application. Py_Initialize should return false before you do so, and true afterwards. If that is not the case (e.g. if Py_Initialize returns true even though you haven't called Py_Initialize), something is wrong. For now, I am testing this in Windows with Visual Studio. I will look into renaming the symbols, but I am not sure I understand why this is necessary. Surely two unrelated DLLs can exist in a process with the same function names? On Windows, yes. Not on Unix, atleast not in general. If not, I am surprised this problem hasn't been seen or even exploited by plugin competitors more often. For plugins, there is a real problem on Unix, with several strategies to solve it available. Avoiding naming conflicts in the first place is the only one that really works, independently of the plugin host. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Python loading library containing embedded python...
Alex Martelli wrote: I believe the python core developers who do develop on Windows have not even tried VS 2005 and have no intention of so doing for the foreseeable future. That is not true. I have already tried it. Many others did, too, and found that Microsoft managed to break ISO C compliance in VS 2005 (in order to make it safe). The Python trunk has a work-around for that problem. Unfortunate -- as a python core developer myself, while I mostly use MacOSX, and secondarily Linux, I'd be happy to lend a hand occasionally with Windows *IF* I could use VS 2005, which I can get for free, rather than any product I'd have to pay for. You can use VS2005 just fine. You have to convert the project files, and then you are done. The choice of what Windows environment is best is obviously up to those python core developers who DO use Windows day-in, day-out Not really. Users formulate all kinds of demands, and we listen to them. Name an arbitrary Windows compiler, and somebody will have proposed to use that as the compiler for the official Windows binaries. Mention that you are going to stick with the compiler you use, and somebody will complain. Mention that you are going to change compilers, and somebody will complain. You can't win. I don't know of any such plans, sadly. You may want to take up the subject on python-dev: few core developers routinely follow this list (or newsgroup, whatever;-). If you volunteer to port the development environment for Windows to X, and can show that the performance with whatever compiler X offers is better than VS's, you may stand a chance. Good luck... you'll need quite a bit;-). I personally don't care much about the performance of the compiler. For my own development, I care about the tool integration, in particular about the debugger (I like the MSVC debugger quite a lot). For the Python project, I care about the compilers people want to use to build extensions with. For example, you can't really build COM-based extensions with cygwin, so if the official compiler would be cygwin, somebody would have to find out how to build PythonWin (in addition, PythonWin also has a strong dependency on MSVCRT). Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Python loading library containing embedded python...
I have a dll/so which embeds python. I can verify it works by compiling it as an executable and adding an appropriate main. I tried to write unit tests for this library with ctypes and a simple python script. Access violations and other strange things result. I suspect this is because I am basically embedding python in python at this point. How can I make my dll/so with embedded python support use via ctypes? If Py_NewInterpreter is the answer, why does it hang indefinitely? The dll/so must support use in processes which might contain other instances of Python. I can not change that requirement. Running via ctypes is an easy test of this capability (or much of it I suspect). -- http://mail.python.org/mailman/listinfo/python-list