Re: [Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability
[SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them. except ... posixmodule gets renamed to ntmodule oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule. thinking about this some more... posixmodule.c is linked (by default) into pythonNN.dll, thus making pythonNN.dll totally dependent on a version of msvcrt. decoupling posixmodule.c from pythonNN.dll leaves the possibility to make python independent of msvcrt versioning. it would need to be a custom-compiled .pyd module, due to the early dependency. i'll see if this is possible. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability
decoupling posixmodule.c from pythonNN.dll leaves the possibility to
make python independent of msvcrt versioning.
it would need to be a custom-compiled .pyd module, due to the early
dependency.
i'll see if this is possible.
i'd added PyExc_OSError, for example, as data exported from dlls. i'm
finding that this causes problems :)
so when posixmodule.c is a module (nt.pyd), doing this works:
PyAPI_FUNC(PyObject *)
PyErr_GetPyExc_OSError(void)
{
return (PyObject*)PyExc_OSError;
}
and thus
oserr = PyErr_GetPyExc_OSError();
Py_INCREF(oserr);
PyModule_AddObject(m, error, oserr)
but doing _direct_ access to PyExc_OSError fails miserably.
i'll try to track down why (am adding __cdecl to PyAPI_DATA to see if
that helps).
l.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] progress: compiling python2.5 under msys (specifically but not exclusively under wine) with msvcr80
Have you made some benchmarks like pystone? Cheers, Cesare Cesare, hi, thanks for responding: unfortunately, there's absolutely no point in making any benchmark figures under an emulated environment which does things like take 2 billion instruction cycles to start up a program named c:/msys/bin/sh.exe, due to it inexplicably loading 200 GUI-only truetype fonts. and to do benchmarks on say windows would require that i install ... windows! so if somebody else would like to make some benchmarks, and publish them, they are most welcome to do so. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microsoft dlls apparently don't support data. implications: PyAPI functions required to access data across modules.
On Sun, Jan 25, 2009 at 3:58 PM, Luke Kenneth Casson Leighton l...@lkcl.net wrote: according to the wikipedia entry on dlls, dlls do not support data, only functions. which would explain two things: 1) why certain modules are forcibly linked into pythonNN.dll 2) why attempts to move them out of pythonNN.dll cause runtime crashes. so i will continue the experiment, and remove all the data references from the pythonNN.def that i added, and deal with the knock-on consequences, which will involve adding get functions. for example, PyAPI_FUNC(char*) _PyStructSequence_Get_UnnamedField(void) use of such functions will allow various bits and pieces - such as PyStructSequence_UnnamedField - to be converted back to static in their respective c files. any objections, speak now, because this will involve quite a bit of work. here is a starting list of data items which will require getter functions, found just by creating a posixmodule.pyd: Info: resolving __Py_NoneStruct by linking to __imp___Py_NoneStruct (auto-import) Info: resolving _Py_FileSystemDefaultEncoding by linking to __imp__Py_FileSystemDefaultEncoding (auto-import) Info: resolving _PyExc_OSError by linking to __imp__PyExc_OSError (auto-import) Info: resolving _PyUnicode_Type by linking to __imp__PyUnicode_Type (auto-import) Info: resolving _PyFloat_Type by linking to __imp__PyFloat_Type (auto-import) Info: resolving _PyExc_TypeError by linking to __imp__PyExc_TypeError (auto-impoModules/posixmodule.ort) Info: resolving _PyExc_RuntimeError by linking to __imp__PyExc_RuntimeError (auto-import) Info: resolving _PyExc_ValueError by linking to __imp__PyExc_ValueError (auto-import) Info: resolving _PyExc_RuntimeWarning by linking to __imp__PyExc_RuntimeWarning (auto-import) Info: resolving _PyExc_NotImplementedError by linking to __imp__PyExc_NotImplementedError (auto-import) obviously, auto-import can't happen. so getter-functions it is. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microsoft dlls apparently don't support data. implications: PyAPI functions required to access data across modules.
here is a starting list of data items which will require getter functions, found just by creating a posixmodule.pyd: Info: resolving __Py_NoneStruct by linking to __imp___Py_NoneStruct (auto-import) by no small coincidence, every single module with which we've had difficulties in the mingw port - _sre, thread, operator, locale, winreg, signal and have been forced to put them into python2N.dll - all of them _happen_ to _directly_ reference the _PyNone_Struct data variable. surpriiise. that means that the Py_None macro must call the getter function. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability
On Sun, Jan 25, 2009 at 3:55 PM, Roumen Petrov
bugtr...@roumenpetrov.info wrote:
Luke Kenneth Casson Leighton wrote:
[SNIP]
Yes it is enough to encapsulate memory allocation and file functions
into
python shared library. The python provide memory allocation functions,
but
not all modules use them. File functions are hiden by posixmodule and
python
modules can't use them.
except ... posixmodule gets renamed to ntmodule oh, i see what
you mean: python modules aren't allowed _direct_ access to msvcrtNN's
file functions, they have to go via posixmodule-renamed-to-ntmodule.
thinking about this some more... posixmodule.c is linked (by
default) into pythonNN.dll, thus making pythonNN.dll totally dependent
on a version of msvcrt.
This is not problem. If python*.dll hide msvcrt and other modules depend
directly from python*.dll I expect issue to be resolved. i.e. python*.dll to
be portable runtime interface.
ye :)
well - it looks like i am having success with removing all references
to data e.g. Py_NoneStruct, all of the PyExc_*Warning and
PyExc_*Error, all of the Py*_Types and more.
i'm making sure that macros are heavily used - so that on systems
where data _can_ be accessed through dynamic shared objects, it's done
so.
#if defined(MS_WINDOWS) || defined(__MINGW32__)
/* Define macros for conveniently creating getter functions,
* to avoid restrictions on dlls being unable to access data.
* see #5056
*/
/* use these for data that is already a pointer */
#define PyAPI_GETHDR(type, obj) \
PyAPI_FUNC(type) _Py_Get_##obj(void);
#define PyAPI_GETIMPL(type, obj) \
PyAPI_FUNC(type) _Py_Get_##obj(void) { return (type)obj; }
#define _PYGET(obj) _Py_Get_##obj()
/* use these for data where a pointer () to the object is returned
* e.g. no longer #define Py_None (Py_NoneStruct) but
* #define Py_None _PTGETPTR(Py_NoneStruct)
*/
#define PyAPI_GETHDRPTR(type, obj) \
PyAPI_FUNC(type) _Py_Get_##obj(void);
#define PyAPI_GETIMPLPTR(type, obj) \
PyAPI_FUNC(type) _Py_Get_##obj(void) { return (type)obj; }
#define _PYGETPTR(obj) _Py_Get_##obj()
#else
/* on systems where data can be accessed directly in shared modules,
* as an optimisation, return the object itself, directly.
*/
#define PyAPI_GETFNIMPL(obj) ;
#define PyAPI_GETHDR(type, obj) ;
#define PyAPI_GETIMPL(type, obj) ;
#define PyAPI_GETIMPLPTR(type, obj) ;
#define _PYGET(obj) (obj)
#define _PYGETPTR(obj) (obj)
#endif /* defined(MS_WINDOWS) || defined(__MINGW32__)*/
as you can see from the Py_None example, on non-dll-based systems, you
get... wow, a macro which returns... exactly what was returned before.
zero impact.
on windows, you end up defining, creating and using a getter
function. two types. one which returns the object, the other returns
a pointer to the object.
l.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] microsoft dlls apparently don't support data. implications: PyAPI functions required to access data across modules.
On Sun, Jan 25, 2009 at 9:01 AM, Matthieu Brucher matthieu.brucher at gmail.com wrote: 2009/1/25 Luke Kenneth Casson Leighton lkcl at lkcl.net: according to the wikipedia entry on dlls, dlls do not support data, only functions. What do you mean by not support data? Having global data variables in a dll? In wikipedia, it is explicitely told that this is possible to have data (http://en.wikipedia.org/wiki/Dynamic-link_library). Without them, shared library cannot be used. matthieu, thank you for responding. from http://en.wikipedia.org/wiki/Dynamic-link_library: Third, dynamic linking is inherently the wrong model for paged memory managed systems. Such systems work best with the idea that code is invariant from the time of assembly/compilation on. ... Data references do not need to be so vectored because DLLs do not share data. does anyone happen to know what this means? also, what do you mean by without data, shared library cannot be used? you can _always_ call a function which returns a pointer to the data, rather than access the data directly. Indeed. That's why the header files contain define PyAPI_DATA(RTYPE) extern __declspec(dllexport) RTYPE define PyAPI_DATA(RTYPE) extern __declspec(dllimport) RTYPE curt, thank you for responding. i'd seen this: i understood it - and... yet... mingw happily segfaults when asked to access _any_ data in _any_ object file of the python2N dll. Py_NoneStruct, PyExc_* (of which there are about 50), Py*_Type - all of them. solutions so far involve ensuring that anything declared with PyAPI_DATA is *NEVER* accessed [across a dll boundary] - by for example moving the module into python2N.dll. also, yes i had looked up how to do .def files, and how __declspec(dllexport) etc. work. of all the examples that you find about dlltool, mingw, dlls, defs, etc. they _all_ say function. to declare a _function_ you do X, Y and Z. not one of them says to place data in a dll, you do X Y and Z. then, looking at the wine dlls and .defs, i haven't yet found a _single_ one which returns data - they're _all_ functions (from looking so far. e.g. i expected MSVCRT.DLL errno to be an int - it's not: it's a function). *sigh*. if this turns out to be yet another gcc / mingw bug i'm going to be slightly annoyed. only slightly, because this _is_ free software, after all :) l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability
On Fri, Jan 23, 2009 at 10:48 PM, Roumen Petrov bugtr...@roumenpetrov.info wrote: python.exe (say, the official one) loads python25.dll. Then, an import is made of a ming-wine extension, say foo.pyd, which is linked with libpython2.5.dll, which then gets loaded. Voila, you have two interpreters in memory, with different type objects, memory heaps, and so on. ok, there's a solution for that - the gist of the solution is already implemented in things like Apache Runtime and Apache2 (modules), and is an extremely common standard technique implemented in OS kernels. the old school name for it is vector tables. [SNIP] Did you think that this will escape python MSVC from Assembly hell ? let me think about that write some things down, i might have an answer at the end :) but it would certainly mean that there would be both a future-proof path for binary modules from either msvc-compiled _or_ mingw-compiled 2.5, 2.6, 2.7 etc. to work with 2.5, 2.6, 2.7, 2.8 etc. _without_ a recompile. [forwards-future-proof-compatibility _is_ possible, but... it's a bit more... complicated. backwards-compatibility is easy]. what you do is you make sure that the vector-table is always and only extended - added to - never removed from or altered. if one function turns out to be a screw-up (inadequate, not enough parameters), you do NOT change its function parameters, you add an Ex version - or an Ex1 version. just like microsoft does. [ now you know _why_ they do that ridiculous thing of adding FunctionEx1 FunctionEx2 and if you look at the MSHTML specification i think they go up to _six_ revisions of the same function in one case!] to detect revisions of the vector-table you use a negotiation tactic. you add a bit-field at the beginning of the struct, and each bit expresses a new revision indicating that the vector table has been extended (and so needs to be typecast to a different struct - exactly the same as is done with PyObject being typecast to different structs). the first _function_ in the vector-table is one which the module must call (in its init()) to pass in the version number of the module, to the python runtime. just in case someone needs to know. but for the most part, the initiation - of function call-out - is done _from_ modules, so each and every module will never try to call something beyond what it understands. but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also it's actually _already_ in use in one form in the way that python objects are typecast from PyObject to other types of structs! the difference is that a bit-field would make detection of revisions a bit easier but to be honest you could just as easily make it an int and increase the revision number. ok, i've thought about your question, and i think it might [save us from assembly hell]. what you would likely have to do is compile _individual modules_ with assemblies, should they need them. for example, the msvcrt module would obviously have to be hey, that'd be interesting, how about having different linked versions of the msvcrt module? coool :) in the mingw builds, it's not necessary to link in PC/msvcrtmodule.o into the python dll - so (and this confused the hell out of me for a minute so i had to do find . -name msvcrt*) you end up with a Modules/msvcrt.pyd. surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc. i'll do an experiment, later, to confirm that this actually _does_ work - i.e. creating an msvcr80.pyd with mingw gcc -specs=msvcr80. the neat thing is that if it works, you wouldn't need to _force_ people to link to the python dll or the python exe with msvcr90 or any other version. and the mingw built python.exe or python dll would be interchangeable, as it would be _specific modules_ that required specific versions of the msvc runtime. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability
but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also This look like simple RPC implemantation. yep. If I remember well SUN-RPC assign number to program, function, version. yep - i forgot about that one: yes, that's another example. this is pretty basic well-understood, well-documented techniques that virtually every large project that requires isolation between components (and an upgrade path) ends up using in one form or another. the only fly in the ointment will be that putting pointers to PyType_String etc. etc. into a vector table (struct), you end up with an extra de-ref overhead, which is often an argument utilised to do away with vector tables. but - tough: the decision involves getting away from Hell to something that makes everyone's lives that much easier, it's an easy decision to make. surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc. [SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them. except ... posixmodule gets renamed to ntmodule oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule. so it's still ok. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] mingw+msys port of python2.7a0 (svn #r68884)
http://bugs.python.org/issue5046 mingw+msys port which was previously done against python-2.5.2 has been brought forward to latest subversion r68884. the primary reason for initially doing python 2.5.2 was to 1) stay out of the way of primary python development 2) provide some hope for those people still using win98 and nt. builds using msvcr90 are possible with the --enable-msvcr9build switch. install mingw, install msys, optionally install a boat-load of libraries such as sqlite3, zlib, bz2, dbm etc. (good luck, it's a pain - ask me if you'd like some prebuilt). make sure you patch the mingw runtime if you want to use --enable-msvcr9build (again, ask me if you'd like the patched headers and prebuilt libmsvcrNN.a files). also added is the msi module - http:/ /lkcl.net/msi.tgz and run make in the msi directory, to install the import libraries and header files borrowed from Wine and beaten into submission. run ./configure --enable-win32build=yes --enable-shared=yes and go do something else for about 10 minutes, then run make and make install. no proprietary compilers or tools were used or harmed [*] in the making or development of this patch. l. [*] such a shame... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Wed, Jan 21, 2009 at 9:13 PM, Martin v. Löwis mar...@v.loewis.de wrote: ok, so - different from what's being generated by ./configure under msys under wine or native win32 - what's being generated (libpython 2 . 5 . a and libpython 2 . 5 . dll . a) is more akin to the cygwin environment. therefore, there's absolutely no doubt that the two are completely different. and on that basis, would i be correct in thinking that you _can't_ go linking or building modules or any python win32 code for one and have a hope in hell of using it on the other, and that you would _have_ to rebuild e.g. numpy for use with a mingw32-msys-built version of python? I can't comment on that, because I don't know what your port does. Does it not produce a .dll containing the majority of Python? no, it contains the minimal necessary amount of python modules, exactly like when python is built using cygwin. actualy, there's a few modules that _have_ to be included. roumen discovered that you have to have these: _functools _functoolsmodule.c# Tools for working with functions and callable objects operator operator.c # operator.add() and similar goodies _locale _localemodule.c # -lintl _struct _struct.c _subprocess ../PC/_subprocess.c _winreg ../PC/_winreg.c and i've discovered that when running under wine you have to also have these: _weakref _weakref.c and also when running unde wine with msvcr80, so far, you have to also have these: collections collectionsmodule.c thread threadmodule.c all the rest can be done as .pyd And is that not called python25.dll? no, it's called libpython2.5.dll.a, just like when python is built using cygwin. the configure scripts, thanks to the cygwin build, already end up copying that to libpython2.5.dll. _not_ python25.dll l. p.s. there's nothing to stop you adding every single module and then renaming the resultant blob to libpython25.dll - i just haven't been given, or found, a good reason to do so yet. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 9:18 AM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: or, is the .pyd loading a bit cleverer (or perhaps a bit less cleverer) than i'm expecting it to be? On Windows, you must turn on the --enable_shared option if you want to build extension modules. You could take the cygwin build as an example, see what's done in ./configure.in. amaury, thank you for mentioning that - yes, as it turns out, all of the mingw ports (dan, roumen etc) do pretty much exactly this. also it turns out that on mingw, if you _don't_ disable shared (i.e. if you try to build a static library) mingw32 gcc runtime utils .16, .17 _and_ .19 all segfault or have runtime assertions when creating the archives!! either ar.exe or ranlib.exe choke themselves to death. which is greaaat. so, i've had to set the variable which specifies the libpython2.5.a static library to in order to stop it from being built. it would be helpful if there was a --enable-static=yes/no configure option, but there isn't one. leaving that aside, you understand therefore that dan, roumen and i have all managed to achieve building of .pyd extension modules. so, the question i am asking is: would it be reasonable to expect mingw-compiled .pyd modules to work with a proprietary-compiled msvc python.exe, and vice-versa? l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 6:43 PM, Martin v. Löwis mar...@v.loewis.de wrote: I can't comment on that, because I don't know what your port does. Does it not produce a .dll containing the majority of Python? no, it contains the minimal necessary amount of python modules, exactly like when python is built using cygwin. actualy, there's a few modules that _have_ to be included. That's actually not my question. ah right - sorry to not quite fully understand. Do you have a DLL that contains all of Python/*.o and Objects/*.o? yes. And is that not called python25.dll? no, it's called libpython2.5.dll.a, just like when python is built using cygwin. the configure scripts, thanks to the cygwin build, already end up copying that to libpython2.5.dll. _not_ python25.dll I'm giving up for now. I don't quite understand why the file gets first called libpython2.5.dll.a, and then renamed to libpython2.5.dll. Could it be perhaps that the .a file is an import library, and actually different from the .dll file? ... *thinks*... sorry, you're right. it's the way that dlltool is used on cygwin. dlltool on cygwin and gcc on cygwin create files with the following equivalence: python25.lib on msvc --- libpython2.5.dll.a on cygwin and mingw32 python2.5.dll on msvc -- libpython2.5.dll on cygwin and mingw32 p.s. there's nothing to stop you adding every single module and then renaming the resultant blob to libpython25.dll - i just haven't been given, or found, a good reason to do so yet. That doesn't really matter, I guess. An extension module build by your port will either fail to load into the regular Python (if libpython2.5.dll is not found), or load and then crash (because it uses a different copy of the Python runtime). Likewise vice versa. If this port ever takes off, we get another binary-incompatible Python version. there are at least three [mingw] already. I hope that users will understand that it is disjoint from the python.org version (as they seem to understand fine for the Cygwin build, which already picks up its extension modules also from a disjoint location, which helps to keep the two separate). there are already no less than _four_ mingw ports of python, of varying degrees. * http://jove.prohosting.com/iwave/ipython/pyMinGW.html * http://sebsauvage.net/python/mingw.html * http://python-mingw.donbennett.org/ * roumen's cross-compile+native port * the port i'm working on - extending roumen's native mingw compile one dates back to... python 2.2 i didn't include that one. another is python2.4. don's work is a cygwin cross-compile (note NOT a compile of python for cygwin such that you need CYGWIN.DLL to run python), so, using cygwin under win32 to cross-compile a native python.exe. smart, that. roumen then worked on that further, to make it compile under mingw / msys, not cygwin. and i'm working on taking windows _completely_ out of the loop, by getting python.exe to both compile _and_ run under wine, with the added benefit that if you _did_ happen to want to compile (or run) under either native windows or both, you can. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
That doesn't really matter, I guess. An extension module build by your port will either fail to load into the regular Python (if libpython2.5.dll is not found), or load and then crash (because it uses a different copy of the Python runtime). Likewise vice versa. excellent, excellent that's _really_ good - and here's why: if it is _guaranteed_ to crash, regardless of what i do (because the copy of the python runtime is different), then it _doesn't_ matter what version of msvcrt i link the mingw-built python runtime with, does it? am i right? and if _that's_ the case, i can stop fricking about with msvcr80 :) which would be an absolute godsend because msvcr80 under wine is a frickin nightmare. the python regression tests pretty much hammer the daylights out of wine and it's squeaking in all sorts of weird places. adding in msvcr80, an undocumented transparent blob into the mix just makes getting this port fully operational all the more difficult. i'd like to avoid that :) l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 8:17 PM, Martin v. Löwis mar...@v.loewis.de wrote: am i right? You should test that. I'm not sure whether it will crash (in particular, it might not on load), but it *might* crash, or fail in strange ways (e.g. when it checks whether something is a string, and decides it is not, because it is looking at the other PyString_Type) mrmmm... how? apps won't end up loading _both_ libpython2.5.dll _and_ python25.dll (or libpython2.N.dll and python2N.dll) will they? and if _that's_ the case, i can stop fricking about with msvcr80 :) If so, I think there is little point in submitting patches to the Python bug tracker. I'm -1 on supporting two different-but-similar builds on Windows. I could accept a different build *process*, but the outcome must be the same either way. (of course, msvcr80 is irrelevant, because Python had never been using that officially) oh? i saw the PCbuild8 and thought it was. oh that's even better - if python2.5 only officially support msvcrt whew. ok , i see - python2.6 uses msvcr90. i'll cross that bridge when i come to it. l. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 8:22 PM, Martin v. Löwis mar...@v.loewis.de wrote: This doesn't seem to be distributing binaries. sourceforge page. i checked the statistics, there don't seem to be very many hits (sorry to hear that don, if you're reading this!) ok. there _is_ a sourceforge page,... yep, downloads here: http://sourceforge.net/project/showfiles.php?group_id=182839 Where *exactly* do you get binaries there? All I can find is patches-2.5.1v2.gz doh! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 9:09 PM, Martin v. Löwis mar...@v.loewis.de wrote: mrmmm... how? apps won't end up loading _both_ libpython2.5.dll _and_ python25.dll (or libpython2.N.dll and python2N.dll) will they? Of course they will! yeah, silly - i worked that out juust after i pressed send. python.exe (say, the official one) loads python25.dll. Then, an import is made of a ming-wine extension, say foo.pyd, which is linked with libpython2.5.dll, which then gets loaded. Voila, you have two interpreters in memory, with different type objects, memory heaps, and so on. ok, there's a solution for that - the gist of the solution is already implemented in things like Apache Runtime and Apache2 (modules), and is an extremely common standard technique implemented in OS kernels. the old school name for it is vector tables. so you never let the .pyd (or so even) modules link to the libpythonN.N.dll, pythonNN.dll (or libpythonN.N.so even), you _pass in_ a pointer to everything it's ever going to need (in its initmodule) function. This was always the problem when an old extension module (say, from 2.4) was loaded into a new interpreter (say, 2.5), then you load both python25.dll and python24.dll, causing crashes. To prevent this issue, Python now checks whether the module is linked with an incorrect pythonxy.dll, but won't catch that libpython2.5.dll is also a VM. ok. i'll look at making libpython2.5.dll equal to python25.dll. (of course, msvcr80 is irrelevant, because Python had never been using that officially) oh? i saw the PCbuild8 and thought it was. oh that's even better - if python2.5 only officially support msvcrt whew. No, Python 2.5 is linked with msvcr71.dll. ehn? i don't see that anywhere in any PC/* files - i do see that there's a dependency on .NET SDK 1.1 which uses msvcr71.dll still, 71 is good news - as long as it's not involving assemblies. 2.6 is a different matter, but, thinking about it, i have hopes that the better-tested-codepath of the 2.6 codebase would have better luck with 9.0 [than i had with 2.5 and 8.0] simply because... it's been tested already! [and 2.5 with 8.0 hadn't] l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Fri, Jan 23, 2009 at 6:36 AM, Luke Kenneth Casson Leighton l...@lkcl.net wrote: On Thu, Jan 22, 2009 at 9:09 PM, Martin v. Löwis mar...@v.loewis.de wrote: mrmmm... how? apps won't end up loading _both_ libpython2.5.dll _and_ python25.dll (or libpython2.N.dll and python2N.dll) will they? Of course they will! yeah, silly - i worked that out juust after i pressed send. ironically, i started out with the intent of going for python2N.dll interoperability. then i noticed that all the other mingw ports dropped the total-inclusion-of-all-modules because you _can_. i should have stuck with my original plan :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Thu, Jan 22, 2009 at 10:16 PM, Mark Hammond skippy.hamm...@gmail.com wrote: On 23/01/2009 7:01 AM, Luke Kenneth Casson Leighton wrote: That doesn't really matter, I guess. An extension module build by your port will either fail to load into the regular Python (if libpython2.5.dll is not found), or load and then crash (because it uses a different copy of the Python runtime). Likewise vice versa. excellent, excellent that's _really_ good - and here's why: if it is _guaranteed_ to crash, regardless of what i do (because the copy of the python runtime is different), then it _doesn't_ matter what version of msvcrt i link the mingw-built python runtime with, does it? I'm very confused about this: It seems you started this work precisely so you can be compatible between MSVC built Python's and mingw builds yeah that's where i _started_ - and after being on this for what nearly eight days straight i was hoping to get away with as little extra work as possible. - ie, this thread starts with you saying: this is a fairly important issue for python development interoperability - but now you seem to be saying it is actually a *feature* if they don't work together? *sigh* no, that was me getting confused and if _that's_ the case, i can stop fricking about with msvcr80 :) If all you are doing is trying to get a version of Python working under Wine that isn't compatible with MSVC built binaries, I can't work out why you are fricking around with msvcr80 either! ha ha :) existence of PCbuild8 is the main reason :) that and getting the wrong end of the stick. i'll get there eventually. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
It only becomes a problem when someone wants to both support Windows users of their extension modules with pre-built binaries, but *also* doesn't want to set up the appropriate environment for building such binaries (currently a minimum bar of Visual Studio Express on a Windows VM instance). ok - fortunately, thanks to dan kegel for pointing me in the right direction of winetricks vcrun2005p1 i was able to get a successful build using Microsoft.VC8.CRT assemblies. i say successful because Parser/pgen.exe was built and ran, and libpython2.5.dll.a was also successfully built, as was python.exe successfully built. the problem _now_ to overcome is that the bloody libmsvcrt80.a has the wrong definitions, for a 32-bit build! it has functions like _fstat instead of _fstat32 and so on. if this was a 64-bit version of wine i was using mingw32 under, i would not have encountered this issue. amazingly, however, someone _else_ who kindly tried out compiling python2.5 with mingw and msvcr80, native on win32, reported that it was a complete success! as in, successful build, successful install, successful run of tests, only 4 failed regression tests. i am utterly mystified as to how that happened. next task: beat the crap out of libmsvcr80.a and /mingw/include/*.h, repeat until success. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Tue, Jan 20, 2009 at 9:19 PM, Martin v. Löwis mar...@v.loewis.de wrote: That's a non-starter for anyone who incorporates Python in an existing MSVC-based development environment. surely incorporating libpython2.5.dll.a or libpython2.6.dll.a, along with the .def and the importlib that's generated with dlldump, unless i'm missing something, would be a simple matter, yes? Not exactly sure what this is, but I believe Python *already* includes such a thing. sorry, martin - i thought the win32 builds generated python25.lib, python25.dll and python25.def so as to fit into the 8.3 filename convention. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
next task: beat the crap out of libmsvcr80.a and /mingw/include/*.h, repeat until success. https://sourceforge.net/tracker/index.php?func=detailaid=2134161group_id=2435atid=352435 roumen, it looks like you've been and done that, already - thank you! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] progress: compiling python2.5 under msys (specifically but not exclusively under wine) with msvcr80
this is a progress report on compiling python using entirely free software tools, no proprietary compilers or operating systems involved, yet still linking and successfully running with msvcr80 assemblies. manifests and rc files, which are compiled to internal resources, have been added. various sections which are uniquely identifed by _MSC_VER = 1400 etc have had to be enabled with corresponding MSVCRT_VERSION = 0x0800 - in particular, signal handling (PyOS_getsig()). currently, under wine with msvcr80, there looks like there is a bug with a common theme related to threads, but here's a short list: test_array.py is blocking, test_bz2.py is hanging and test_cmd_line.py causes a segfault; test_ctypes is _still_ a bundle of fun. for those people who use native win32 platforms who are compiling up this code, you should have better luck. significantly, the wine developers have been absolutely fantastic, and have fixed several bugs in wine, sometimes within hours, that were found as a result of running the extremely comprehensive python regression tests. the python regression tests are a credit to the collaborative incremental improvement process of free software development. i look forward to seeing the same incremental improvement applied to the development of python, evidence of which would be clearly seen by the acceptance of one of the following patches, one of which is dated 2003: http://bugs.python.org/issue3754 http://bugs.python.org/issue841454 http://bugs.python.org/issue3871 http://bugs.python.org/issue4954 http://bugs.python.org/issue5010 for those people wishing to track and contribute to the development of python for win32 using entirely free software tools, either under wine or native windows, there is a git repository, here, slightly illogically named pythonwine because that's where i started from (cross-compiling python under wine, so i could get at the wine registry from python). obviously, since then, things have... moved on :) http://github.com/lkcl/pythonwine/tree/python_2.5.2_wine l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] progress: compiling python2.5 under msys (specifically but not exclusively under wine) with msvcr80
http://bugs.python.org/issue5010 correction: that's http://bugs.python.org/issue5026 apologies for the mix-up. also,for the msvcrt80 build, it is _essential_ that you use a patched version of mingw32-runtime, see: https://sourceforge.net/tracker/index.php?func=detailaid=2134161group_id=2435atid=352435 libmsvcr80.a mistakenly thinks that _fstat exists (it doesn't - only _fstat32 does, and many more). it's quite straightforward to rebuild - just remember to run ./configure --prefix=/mingw and if you want to revert just reinstall mingw runtime .exe l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Wed, Jan 21, 2009 at 7:42 PM, Martin v. Löwis mar...@v.loewis.de wrote: sorry, martin - i thought the win32 builds generated python25.lib, python25.dll Correct. and python25.def No. so as to fit into the 8.3 filename convention. No. It generates python25.lib because that's the import library for python25.dll. It calls it python25.dll because the lib prefix is atypical for the platform, and also redundant (DLL means dynamic link library). The Python binary installer also includes libpython25.a, for use with mingw32. ok, so - different from what's being generated by ./configure under msys under wine or native win32 - what's being generated (libpython 2 . 5 . a and libpython 2 . 5 . dll . a) is more akin to the cygwin environment. therefore, there's absolutely no doubt that the two are completely different. and on that basis, would i be correct in thinking that you _can't_ go linking or building modules or any python win32 code for one and have a hope in hell of using it on the other, and that you would _have_ to rebuild e.g. numpy for use with a mingw32-msys-built version of python? or, is the .pyd loading a bit cleverer (or perhaps a bit less cleverer) than i'm expecting it to be? l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
folks, hi, this is a fairly important issue for python development interoperability - martin mentioned that releases of mingw-compiled python, if done with a non-interoperable version of msvcrt, would cause much mayhem. well, compiling python on mingw with msvcr80 _can_ be done; using it can also be a simple matter of creating a python.exe.manifest file, but i can't actually do any testing because it doesn't work under wine. so, pending any further advice and guidance from anyone which allows me to successfully proceed, i'm not going to continue to compile - or release - python2.5 *or* python2.6 builds (when i get round to that) using msvcr80 or msvcr9X. one issue in favour of this decision is that the DLL that's produced by the autoconf build process is libpython2.5.dll.a - not python2.5.dll. it has a different name. it should be abundantly clear to users and developers that if name equals libpython2.5.dll.a then duh build equals different. additionally, the setup.py distutils all goes swimmingly well and lovely - using libpython2.5.dll.a. the only issue which _is_ going to throw a spanner in the works is that people who download win32-built precompiled c-based modules are going to find that hey, it don't work! and the answer will have to be go get a version of that module, compiled with mingw, not MSVC. of course - if python for win32 ENTIRELY DROPPED msvc as a development platform, and went for an entirely free software development toolchain, then this problem goes away. thoughts, anyone? l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 (msys+mingw+wine) - giving up using msvcr80 assemblies for now
On Tue, Jan 20, 2009 at 1:11 PM, Tim Lesher tles...@gmail.com wrote: On Tue, Jan 20, 2009 at 08:02, Luke Kenneth Casson Leighton luke.leigh...@googlemail.com wrote: of course - if python for win32 ENTIRELY DROPPED msvc as a development platform, and went for an entirely free software development toolchain, then this problem goes away. That's a non-starter for anyone who incorporates Python in an existing MSVC-based development environment. surely incorporating libpython2.5.dll.a or libpython2.6.dll.a, along with the .def and the importlib that's generated with dlldump, unless i'm missing something, would be a simple matter, yes? When in Rome... yeah they said the same thing about gas ovens, too. not the nazi gas ovens, the phrase my mother used to say if someone stuck their head in a gas oven, would you do the same?. There would also be a significant performance cost. The PGO (Profile Guided Optimisation) compilation of Visual Studio is impressive. i'd say great - but given a choice of impressive profile guided optimisation plus a proprietary compiler, proprietary operating system _and_ being forced to purchase a system _capable_ of running said proprietary compiler, said proprietary operating system, _and_ giving up free software principles _and_ having to go through patch-pain, install-pain _and_ being forced to use a GUI-based IDE for compilation or free software tools and downloads the use of which means i am beholden to NOONE, it's a simple choice for me to make - maybe not for other people. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] one last go at msvcr80 / msvcr90 assemblies - mingw build of python
could someone kindly send me the assembly files that are created by a proprietary win32 build of python2.5, 2.6 and trunk, the ones used to create the dll _and_ the python.exe many thanks. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] compiling python2.5 (msys+mingw+wine) using msvcr80 assemblies
folks, hi, after some quiet advice i've tracked down a method for compiling python2.5 using msvcr80 that _will_ actually work both under native win32 and also under wine, but it's a _bit_ dodgy, as i couldn't track down where you're supposed to put Microsoft.VC80.CRT, except in the path of the application where it's running from. so, instead, i put the _contents_ of Microsoft.VC80.CRT.manifest into the manifest for the file, and this _does_ actually seem to work. i'm thinking of adding the Microsoft.VC80.CRT.manifest to the rc file (for compilation as a resource) to see if _that_ works, and will report back, but first i wanted to describe what i've done and see what people think: 1) created python_2.5_8.0_mingw_exe.manifest contents as follows, this is _normally_ what is in Microsoft.VC80.CRT _not_ in the .exe.manifest ?xml version='1.0' encoding='UTF-8' standalone='yes'? assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0' file name=msvcr80.dll hash=10f4cb2831f1e9288a73387a8734a8b604e5beaa hashalg=SHA1asmv2:hash xmlns:asmv2=urn:schemas-microsoft-com:asm.v2 xmlns:dsig=http://www.w3.org/2000/09/xmldsig#;dsig:Transformsdsig:Transform Algorithm=urn:schemas-microsoft-com:HashTransforms.Identity/dsig:Transform/dsig:Transformsdsig:DigestMethod Algorithm=http://www.w3.org/2000/09/xmldsig#sha1;/dsig:DigestMethoddsig:DigestValuen9On8FItNsK/DmT8UQxu6jYDtWQ=/dsig:DigestValue/asmv2:hash/file file name=msvcp80.dll hash=b2082dfd3009365c5b287448dcb3b4e2158a6d26 hashalg=SHA1asmv2:hash xmlns:asmv2=urn:schemas-microsoft-com:asm.v2 xmlns:dsig=http://www.w3.org/2000/09/xmldsig#;dsig:Transformsdsig:Transform Algorithm=urn:schemas-microsoft-com:HashTransforms.Identity/dsig:Transform/dsig:Transformsdsig:DigestMethod Algorithm=http://www.w3.org/2000/09/xmldsig#sha1;/dsig:DigestMethoddsig:DigestValue0KJ/VTwP4OUHx98HlIW2AdW1kuY=/dsig:DigestValue/asmv2:hash/file file name=msvcm80.dll hash=542490d0fcf8615c46d0ca487033ccaeb3941f0b hashalg=SHA1asmv2:hash xmlns:asmv2=urn:schemas-microsoft-com:asm.v2 xmlns:dsig=http://www.w3.org/2000/09/xmldsig#;dsig:Transformsdsig:Transform Algorithm=urn:schemas-microsoft-com:HashTransforms.Identity/dsig:Transform/dsig:Transformsdsig:DigestMethod Algorithm=http://www.w3.org/2000/09/xmldsig#sha1;/dsig:DigestMethoddsig:DigestValueYJuB+9Os2oxW4mY+2oC/r8lICZE=/dsig:DigestValue/asmv2:hash/file /assembly 2) created python_2.5_8.0_mingw_exe.rc contents as follows: #include winuser.h 2 RT_MANIFEST PC/python_2.5_8.0_mingw_exe.manifest you could get away with 2 24 PC/. and could exclude the #include 3) added a rule to Makefile.pre.in to create the .res as a binary: # This rule builds the .res file for the Python EXE, required when # linking and using msvcrt80 or above. good luck to us all... $(PYTHONEXEMSVRES): $(srcdir)/PC/python_$(VERSION)_$(MSRTVER)_exe.manifest \ $(srcdir)/PC/python_$(VERSION)_$(MSRTVER)_mingw_exe.rc windres --input $(srcdir)/PC/python_$(VERSION)_$(MSRTVER)_mingw_exe.rc \ --output $(PYTHONEXEMSVRES) \ --output-format=coff 4) added $(PYTHONEXEMSVRES) to the objects to be linked. stunningly, this actually works (of course, you need an msvcr80.dll for it to work duh). i tried finding a location to place the Microsoft.VC80.CRT.Manifest, prior to this hack - a wine dump showed this: 0009:trace:actctx:lookup_assembly looking for name=LMicrosoft.VC80.CRT version=8.0.50727.762 arch=Lx86 0009:trace:heap:RtlAllocateHeap (0x11,0002,0038): returning 0x115148 0009:trace:file:RtlDosPathNameToNtPathName_U (LC:\\windows\\winsxs\\manifests,0xff8c7d08,(nil),(nil)) 0009:trace:file:RtlGetFullPathName_U (LC:\\windows\\winsxs\\manifests 520 0xff8c79f4 (nil)) attempts to copy the manifest into that directory resulted in no joy. so, i'm a bit stuck, and would appreciate some advice on whether the above is acceptable (yes i know it makes sure that python.exe can only use one _very_ specific version of msvcr80.dll - and there are currently two: 8.0.50727.762 and 8.0.50608.0. also i'd appreciate some advice on what the _really_ best way to do this is. and on where the hell i'm supposed to put the VC80.CRT manifest so it will actually... _do_ something! l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] report on building of python 2.5.2 under msys under wine on linux.
hiya roumen, good to hear from you - i've been merging in the work that you did, on mingw native-and-cross compiles. got a couple of questions, will post them in the bugreport ok? On Sat, Jan 17, 2009 at 3:16 PM, Roumen Petrov bugtr...@roumenpetrov.info wrote: Luke Kenneth Casson Leighton wrote: [SNIP] i'm going to _try_ to merge in #3871 but it's... the prospect of sitting waiting for configure to take THREE hours to complete, due to /bin/sh.exe instances taking TWO SECONDS _each_ to start up does not really fill me with deep joy. As from version 1.1.8 msys bash could be run in wine. ah, that might be worth trying. thank you! it's all a bit odd - it still feels like things are being cross-compiled... but they're not... it's just that setup.py has paths that don't _quite_ match up with the msys environment... You could use CPPFLAGS and LDFLAGS to point other locations. Usually wine drive Z: is mapped to filesystem root and visible as /z in msys. oh don't get me wrong - it all works, all compiles absolutely fine, modules and everything... it just... _feels_ like a cross-compile, because of msys. you're sort-of in unix-land, yet the resultant binary python.exe is most _definitely_ c:/ey. the regression testing is _great_ fun! some of the failures are really quite spectacular, but surprisingly there are less than anticipated. About 5 test fail in emulated environment due bugs in emulator. i have 12 that fail - but if i replace msvcrt builtin with msvcrt native, all but two or three disappear. the S8I one (which is due to gcc, you already found that i noticed); tmpfile() tries to write to z:/ which is of course / - the root filesystem; and os.environ['HELLO'] = 'World; os.popen(/bin/sh echo $HELLO).read() != 'World' but i'm not sure if that one _should_ succeed: the only reason it got run is because i _happened_ to have msys installed. May be some math tests fail. Python math tests are too strict for msvcrt 7.0/6.0 (default runtime). Simple work around is to use long double functions from mingwex library. oh i noticed in http://bugs.python.org/issue3871 that you replaced some of the math functions. Other work-around is to build whole python (including modules) with msvcrt 9.0 but I'm not sure that I will publish soon technical details about this build as DLL hell is replaces by Assembly hell (see python issues related to vista and MSVC build). ooo, scarey. oh! roumen, very important: martin says we have to be _really_ careful about releases of mingw-compiled python.exe and libpython2.N.dll - they _have_ to match up with the python-win32 build msvcrt otherwise it will cause absolute mayhem. i'm currently adding an msvcr80 specfile so i can build under wine - see http://bugs.python.org/msg79978 which requires that you customise wine to get it to drop something blah blah don't care in the least, i just want it all to work :) l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] report on building of python 2.5.2 under msys under wine on linux.
About 5 test fail in emulated environment due bugs in emulator. oh - nearly forgot: several of the ctypes tests fail quite spectacularly :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] off-by-one on ftell on wine, but no regression test to catch it
folks, hi, http://bugs.winehq.org/show_bug.cgi?id=16982 related to this: from array import array TESTFN = testfile.txt def fail(x): print x testlines = [ spam, spam and eggs\n, eggs, spam, ham and spam\n, saussages, spam, spam and eggs\n, spam, ham, spam and eggs\n, spam, spam, spam, spam, spam, ham, spam\n, wonderful spaam.\n ] try: # Prepare the testfile bag = open(TESTFN, w) bag.writelines(testlines) bag.close() f = open(TESTFN) testline = testlines.pop(0) line = f.readline() testline = testlines.pop(0) buf = array(c, \x00 * len(testline)) f.readinto(buf) testline = testlines.pop(0) print length of testline:, len(testline) line = f.read(len(testline)) if line != testline: fail(read() after next() with empty buffer failed. Got %r, expected %r % (line, testline)) lines = f.readlines() if lines != testlines: fail(readlines() after next() with empty buffer failed. Got %r, expected %r % (line, testline)) f.close() finally: os.unlink(TESTFN) which is a reduced version of Lib/test/test_file.py running under wine, ftell() has an off-by-one bug, where the file position accidentally doesn't include the fact that the CR of the CRLF has been skipped. but, now with the fgets() bug fixed, the regression tests pass, but there's still the off-by-one bug which _isn't_ caught. this really should be added as a windows test. actually, it should be added as a test for everything: it's not always reasonable to assume that OSes get their file positions right :) l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] http://bugs.python.org/issue4977 - assumption that maxint64 fits into long on 32-bit systems
this was found as part of the regression tests, compiling python under wine under msys with mingw32. test_maxint64 failed. i tracked it down to the assumption that a long will fit into 32-bits, which obviously... it won't! the simplest case is to add a test for the length of the data string being 10 or more characters, on the basis that 9 or less is definitely going into an int; 11 or more is _definitely_ going into a long, and... well... bugger the 10 case, just stuff it in a long rather than waste time trying to find out how many chars it is. if anyone wants to do a more perfect version of this, simple patch, they're mooore than welcome. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] report on building of python 2.5.2 under msys under wine on linux.
no, the above subject-line is not a joke: i really _have_ successfully built python2.5.2 by installing wine on linux, then msys under wine, and then mingw32 compiler - no, not the linux mingw32-cross-compiler, the _native_ mingw32 compiler that runs under msys, and then hacking things into submission until it worked. issue-report: http://bugs.python.org/issue4954 source code: http://github.com/lkcl/pythonwine/tree/python_2.5.2_wine related issue-report: http://bugs.python.org/issue3871 related issue-report: http://bugs.python.org/issue1597850 i'm going to _try_ to merge in #3871 but it's... the prospect of sitting waiting for configure to take THREE hours to complete, due to /bin/sh.exe instances taking TWO SECONDS _each_ to start up does not really fill me with deep joy. consequently i did a major hatchet-job on configure.in with repeated applications of if test $win32build = no; then ... cue several hundred lines of configure.in tests blatantly ignored fi # $win32build=no! and thus cut the configure time down from three hours to a mere 15 minutes. the only reason why this was possible at all was because PC/config.h already exists and has been pre-set-up with lots of lovely #defines. also, there is another significant difference between #3871 and #4954 - i chose to build in to libpython2.5.dll exactly as many modules as are in the proprietary win32 build. this turned out to be a good practical decision, due to /bin/sh.exe messing around and stopping python.exe from running! (under cmd.exe it's fine. i have to do a bit more investigation: my guess is that the msys remounter is getting in the way, somehow. compiling python to have a prefix of /python25 results in files being installed in /python25 which maps to c:/msys/python25/ but actually that doesn't get communicated correctly to the compiled python.exe it's all a bit odd - it still feels like things are being cross-compiled... but they're not... it's just that setup.py has paths that don't _quite_ match up with the msys environment... needs work, there. the regression testing is _great_ fun! some of the failures are really quite spectacular, but surprisingly there are less than anticipated. file sharing violation isn't a big surprise (under wine); the ctypes structure failures are going to be a bitch to hunt down; the test_str %f failure _was_ a big surpise; the builtin file \r\n - \n thing wasn't in the _least_ bit of a surprise :) overall, this has been... interesting. and the key thing is that thanks to #3871 and #4954 and #1597850, python will soon happily compile for win32 _without_ the dependence on _any_ proprietary software or operating systems. that's a pretty significant milestone. l. p.s. if anyone would like to try out this build, on a windows box, to see if it fares any better on the regression tests please say so and i will make the binaries available. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] report on building of python 2.5.2 under msys under wine on linux.
practical decision, due to /bin/sh.exe messing around and stopping python.exe from running! (under cmd.exe it's fine. i have to do a bit more investigation: http://bugs.python.org/issue4956 found it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [patch] moving file load to AFTER Py_Initialize performed
http://bugs.python.org/issue4956 uhn... weird bug, totally left-field scenario (using python under msys under wine under linux) but this rather strange scenario has a situation where loading the filename from the command line cannot be done until _after_ PyInitialize is called. prior to PyInitialize() being called, file handling including stderr is _so_ screwed that it's impossible to even use sprintf(stderr, help help there's something wrong!\n) to diagnose the problem. i wanted to ask people: does this patch (in #4956) look... reasonable? there's no global variables or subtle interaction that steals argv or argc (as part of PyInitialize()) is there, that would, once it's finished, disrupt the loading of the file? if so, it would be necessary to split up the code that gets the filename (filename = argv[_PyOS_optind]) and leave that _bfore_ PyInitialize(), from the code that _loads_ the file (into fp). following the deployment of this fix, the build of python under msys+wine+linux proceeds _from_ msys, rather than having to be irritatingly interrupted and continued from wineconsole cmd. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 on linux under wine
On Thu, Jan 8, 2009 at 9:07 PM, Martin v. Löwis mar...@v.loewis.de wrote: i'd just ... much rather be completely independent of proprietary software when it comes to building free software. I guess my question is then: why do you want to use Windows in the first place? ha ha :) the same question was asked when i started the nt domains reverse-engineering for samba, in 1996. the answer is: i don't. but there are a lot of users and developers who feel that they don't have a choice. or haven't been given one. so if it's possible for me, as one of the under 1% of computer users i.e. linux to compile stuff that will work on the over 95% of computers used by everyone else i.e. windows _and_ i get to stick to free software principles, that's gotta be good. take pywebkit-gtk as an example. the first-level (and some of the second-level) dependencies for pywebkit-gtk are roughly as follows: * libstdc++ * cairo, pango, gdk, fontconfig, gtk * libxml2 (which is dodgy) * libxslt1 (which is so dodgy and dependent on incompatible versions of libxml2 it can't be compiled on win32) * libicu38 * libcurl * libssl * webkit * python2.5 * python-gobect * python-gtk that's a *big* ing list that comes in at a whopping 40mb of _binaries_. webkit itself comes in at 10mb alone. libicu38 fails _miserably_ to cross-compile with mingw32. i was damn lucky to have beaten it into submission: it took two days and i couldn't run any of the tests, but actually managed to get at least some .libs, .dlls and .a's out of the mess. libxslt1 and libxml2 have compile errors in mutually incompatible versions on win32, plus, unfortunately, the versions that _do_ compile correctly (really old versions like libxslt-1.12 + libxml2-18 or something) are not the ones that can be used on webkit! i had to get the source code for gcc (4.4) because when linking webkit against the MSVC-compiled libicu38 gcc actually segfaulted (!). and that was tracked down to exception handling across process / thread boundaries in libstdc++-6 which had only literally been fixed/patched a few days before i started the monster-compile-process. i tried hunting down python-gobject and python-gtk for win32, but there is a dependency needed before you get to that: python25.lib. as i mentioned previously i tried hunting down a .lib for python25 but of course that would be useless unless i also have a libtool-compiled .a so there wasn't any point. so, all the hard work that i did cross-compiling up webkit for win32 was completely wasted because python itself could not be compiled on linux for a win32 platform. hence my interest in making sure that it can be. _then_ i can go back and revisit the monster compile process and finally come up with the goods, on win32, on the gobject-based DOM-model manipulation stuff i've added to pywebkit-gtk. i've got linux covered, i've got macosx covered. win32 is the last one. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 on linux under wine
On Thu, Jan 8, 2009 at 12:42 PM, Simon Cross hodgestar+python...@gmail.com wrote: On Sat, Jan 3, 2009 at 11:22 PM, Luke Kenneth Casson Leighton l...@lkcl.net wrote: secondly, i want a python25.lib which i can use to cross-compile modules for poor windows users _despite_ sticking to my principles and keeping my integrity as a free software developer. If this eventually leads to being able to compile Python software for Windows under Wine (using for example, py2exe) it would make my life a lot easier. that looks like being an accidental side-effect, yes. where i'm up to so far: * i'm using -I $(src_dir)/PC at the beginning of the includes, so that PC/pyconfig.h gets pulled in as a priority over-and-above the auto-generated pyconfig.h (yukkk - i know); this makes the job of building almost-exactly-like-the-visual-studio-build much easier. * i'm manually compiling-linking the Modules/*.c and PC/*modules.c as i also pulled in PC/config.c and left out Modules/config.c - that got me even further * as a result i've actually got a python.exe.so that damnit, it works! the winreg test actually passes for example! the fly in the ointment i'm presently trying to track down: len([1,2]) returns 1L which of course screws up sre_parse.py at line 515 with TypeError: __nonzero__ should return an int because duh if subpattern is returning a Long not an Int. tracking this down further, it would appear that there's some lovely logic in PyInt_FromSsize_t() which i believe is what's getting called from PyInt_AsSsize_t() which is what's getting called from slot_sq_length() (i think) - and, although in this case this build is _definitely_ returning a Long type when it shouldn't, if the value is ever over LONG_MAX then the result will be if subpattern will definitely fail. but... i mean... if ever anyone passes in over 2^^31 items into sre_parse then they _deserve_ to have their code fail, but that's not the point. anyway, i'm floundering around a bit and making a bit of a mess of the code, looking for where LONG_MAX is messing up. l. which of course means that there's a bug in ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 on linux under wine
On Thu, Jan 8, 2009 at 1:11 PM, David Cournapeau courn...@gmail.com wrote: On Thu, Jan 8, 2009 at 9:42 PM, Simon Cross hodgestar+python...@gmail.com wrote: On Sat, Jan 3, 2009 at 11:22 PM, Luke Kenneth Casson Leighton l...@lkcl.net wrote: secondly, i want a python25.lib which i can use to cross-compile modules for poor windows users _despite_ sticking to my principles and keeping my integrity as a free software developer. If this eventually leads to being able to compile Python software for Windows under Wine (using for example, py2exe) it would make my life a lot easier. You can already do that: just install windows python under wine. i tried that a few months ago - the builder requires the MS installer, which segfaulted on my installation of wine (i installed it using winetricks) which left me flummoxed because other people report successful use of MSI. i also don't want just the python.exe, i want the libpython25.a, i want the libpython25.lib, so as to be able to build libraries such as pywekbit-gtk for win32 (cross-compiled using winegcc of course) unpacking the python installer .exe (which was, again, created with a proprietary program) i found that all of the contents were name-mangled and so were useless: i wasn't about to work my way through nearly a hundred files, manually, when i can just as well get python compiling under wine once and then stand a good chance of being able to repeat the exercise in the future, also for python 2.6. so, basically, i really don't want to use visual studio, i really don't want to install a proprietary MSI installer, i really don't want a proprietarily-built python25.exe, and i really don't want a proprietarily-packed installation. i'd just ... much rather be completely independent of proprietary software when it comes to building free software. onwards :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 on linux under wine
anyway, i'm floundering around a bit and making a bit of a mess of the
code, looking for where LONG_MAX is messing up.
fixed with this:
PyObject *
PyInt_FromSsize_t(Py_ssize_t ival)
{
if ((long)ival = (long)LONG_MIN (long)ival = (long)LONG_MAX)
{
return PyInt_FromLong((long)ival);
}
return _PyLong_FromSsize_t(ival);
}
raised as http://bugs.python.org/issue4880
next bug: distutils.sysconfig.get_config_var('srcdir') is returning None (!!)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] compiling python2.5 on linux under wine
next bug: distutils.sysconfig.get_config_var('srcdir') is returning None (!!)
ok ... actually, that's correct. oops.
sysconfig.get_config_vars() only returns these, on win32:
{'EXE': '.exe', 'exec_prefix': 'Z:\\mnt\\src\\python2.5-2.5.2',
'LIBDEST': 'Z:\\mnt\\src\\python2.5-2.5.2\\Lib', 'prefix':
'Z:\\mnt\\src\\python2.5-2.5.2', 'SO': '.pyd', 'BINLIBDEST':
'Z:\\mnt\\src\\python2.5-2.5.2\\Lib', 'INCLUDEPY':
'Z:\\mnt\\src\\python2.5-2.5.2\\include'}
... nd, that means disabling setup.py or hacking it significantly
to support a win32 build, e.g. to build pyexpat, detect which modules
are left, etc. by examining the remaining vcproj files in PCbuild.
ok - i'm done for now.
the project's not complete, but can be regarded as successful so far.
i think the best thing is being able to do import _winreg on a
linux system. that absolutely tickles me silly :)
been running a few tests - test_mmap.py is a hoot, esp. the Try
opening a bad file descriptor... that causes a wine segfault.
if anyone wants to play with this further, source is here:
http://github.com/lkcl/pythonwine/tree/python_2.5.2_wine
at some point - if i feel like taking this further, and if people
offer some advice and hints on where to go (with e.g. setup.py) i'll
continue.
then once that's done i'll do python 2.6 as well.
l.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] compiling python2.5 on linux under wine
hey, has anyone investigated compiling python2.5 using winegcc, under wine? i'm presently working my way through it, just for kicks, and was wondering if anyone would like to pitch in or stare at the mess under a microscope. it's not as crazed as it sounds. cross-compiling python2.5 for win32 with mingw32 is an absolute miserable bitch of a job that goes horribly wrong when you actually try to use the minimalist compiler to do any real work. so i figured that it would be easier to get python compiled using wine. i _have_ got some success - a python script and a python.exe.so (which is winegcc's friendly way of telling you you have something that stands a chance of working) as well as a libpython25.dll.so. what i _don't_ yet have is an _md5.dll (or should it be _md5.lib?) i.e. the standard modules are a bit... iffy. the _winreg.o is compiled; the _md5.o is compiled; the winreg.lib is not. whoops. plus, it's necessary to enable nt_dl.c which is in PC/ _not_ in Modules/. one of the key issues that's a bit of a bitch is that python is compiled up for win32 with a hard-coded pyconfig.h which someone went to a _lot_ of trouble to create by hand instead of using autoconf. oh - and it uses visualstudio so there's not even a Makefile. ignoring that for the time-being was what allowed me to get as far as actually having a python interpreter (with no c-based modules). so there's a whole _stack_ of stuff that needs dragging kicking and screaming into the 21st century. there _is_ a reason why i want to do this. actually, there's two. firstly, i sure as shit do _not_ want to buy, download, install _or_ run visual studio. i flat-out refuse to run an MS os and visual studio runs like a dog under wine. secondly, i want a python25.lib which i can use to cross-compile modules for poor windows users _despite_ sticking to my principles and keeping my integrity as a free software developer. thirdly i'd like to cross-compile pywebkitgtk for win32 fourthly i'd like to compile and link applications to the extremely successful and well wicked MSHTML.DLL... in the _wine_ project :) not the one in windows (!) i want to experiment with DOM model manipulation - from python - similar to the OLPC HulaHop project - _but_ i want to compile or cross-compile everything from linux, not windows (see 1 above) fifthly i'd like to see COM (DCOM) working and pywin32 compiled and useable under wine, even if it means having to get a license to use dcom98 and oleauth.lib and oleauth.h etc. and all the developer files needed to link DCOM applications under windows. actually what i'd _really_ like to see is FreeDCE's DCOM work actually damn well finished, it's only been eight years since wez committed the first versions of the IDL and header files, and it's only been over fifteen years since microsoft began its world domination using COM and DCOM. ... but that's another story :) so that's ... five reasons not two. if anyone would like to collaborate on a crazed project with someone who can't count, i'm happy to make available what i've got up to so far, on github.org. l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] preliminary findings of compiling python to javascript and running it under google V8
http://groups.google.com/group/pyjamas-dev/browse_thread/thread/5e14ac70508112e5/53ca0b8190f35e21?hl=en#53ca0b8190f35e21 may be of interest to some people. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] http://bugs.python.org/issue708007 - telnetpopen
my apologies, i found the issue closed after doing a search for my own work, requiring (again) the telnetlib code for yet another project. since 2001, the work done has been reimplemented time and time again, by different individuals, in different projects, in different ways, all solving exactly the same issues, providing exactly the same type of functionality, as that provided by TelnetPopen3. e.g. Pexpect is the one i found today, listed off of http://en.wikipedia.org/wiki/Expect l. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
