[issue10914] Python sub-interpreter test
Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Éric Araujo mer...@netwok.org added the comment: Except that it didn't work under Windows... Ah. If it was an extension module, I’d have a clue (the debug thing), but I know nothing about embedding. An alternative would be to install the file with the msi system and use a project file to have it compiled (sounds like much hassle). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Ah. If it was an extension module, I’d have a clue (the debug thing), but I know nothing about embedding. An alternative would be to install the file with the msi system and use a project file to have it compiled (sounds like much hassle). Agreed, but it's not something that I know how to do :) (and yet, of course, it would be good to test that embedding works under Windows too, as well as under Unix installs) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: It is possible to install the _testembed.c file alongside xxmodule.c and compile it at test time. We probably could, using the approach from the distutils patch. But it feels really quirky to me. Granted, the Makefile is quirky as well. To test under Windows, I’ve re-read your first patch using distutils and it’s not that long or hard to read after all. Except that it didn't work under Windows... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Éric Araujo mer...@netwok.org added the comment: BTW, about this comment in the code: # XXX only tested under Unix checkouts It is possible to install the _testembed.c file alongside xxmodule.c and compile it at test time. To test under Windows, I’ve re-read your first patch using distutils and it’s not that long or hard to read after all. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Roundup Robot devnull@devnull added the comment: New changeset 5a983773c09a by Victor Stinner in branch '3.2': Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0) http://hg.python.org/cpython/rev/5a983773c09a New changeset 2caf82aee7a4 by Victor Stinner in branch '3.2': Issue #10914: Initialize correctly the filesystem codec when creating a new http://hg.python.org/cpython/rev/2caf82aee7a4 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
STINNER Victor victor.stin...@haypocalc.com added the comment: Victor, the fix needs to go into 3.2 as well. Oh, I thought that Modules/_testembed.c was only in 3.3. Anyway, it is a real bug in 3.2, and it is now fixed (I backported the 2 fixes in 3.2). -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Victor, the fix needs to go into 3.2 as well. -- assignee: - haypo status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Roundup Robot devnull@devnull added the comment: New changeset c8338cfa3578 by Victor Stinner in branch 'default': Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0) http://hg.python.org/cpython/rev/c8338cfa3578 New changeset d3af2a2b621b by Victor Stinner in branch 'default': Issue #10914: Initialize correctly the filesystem codec when creating a new http://hg.python.org/cpython/rev/d3af2a2b621b -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
STINNER Victor victor.stin...@haypocalc.com added the comment: test_capi pass on x86 debian parallel 3.x: I close this issue again :-) -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Roundup Robot devnull@devnull added the comment: New changeset d195ff5c44f4 by Antoine Pitrou in branch '3.2': Issue #10914: Add a minimal embedding test to test_capi. http://hg.python.org/cpython/rev/d195ff5c44f4 New changeset 77cf9e4b144b by Antoine Pitrou in branch '3.2': Issue #10914: add NEWS item. http://hg.python.org/cpython/rev/77cf9e4b144b New changeset ef1ad39e3c40 by Antoine Pitrou in branch 'default': Issue #10914: Add a minimal embedding test to test_capi. http://hg.python.org/cpython/rev/ef1ad39e3c40 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Should be fixed. -- resolution: - fixed stage: - committed/rejected status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: There's a failure on certain locales that Victor, I believe, is currently investigating. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
STINNER Victor victor.stin...@haypocalc.com added the comment: Modules/_testembed fails with ISO-8859-15 locale encoding because of a bootstrap issue. The problem is that the filesystem encoding codec is implemented in Python: Python requires the codec to loads modules, but it has to load a module to load the codec. I already solved this issue in Python using C functions instead of the Python codec until Python is able to load modules (able to load the Python codec). The problem is the test on the python initialization: PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize check the global Py_FileSystemDefaultEncoding variable (use C functions if it is NULL). We should use C functions if Py_FileSystemDefaultEncoding is NULL *or* if interp-codecs_initialized is zero. I think that Python used interp-codecs_initialized flag. I removed the test on interp-codecs_initialized because I thought that it was useless. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
STINNER Victor victor.stin...@haypocalc.com added the comment: I think that Python used interp-codecs_initialized flag Yes in PyUnicode_AsEncodedString(), in Python 3.0 and 3.1. Python 3.2 has also the test, but PyUnicode_AsEncodedString() is no more used to encode filenames. I removed the test in Python 3.3. Extract of Python 3.1: /* During bootstrap, we may need to find the encodings package, to load the file system encoding, and require the file system encoding in order to load the encodings package. Break out of this dependency by assuming that the path to the encodings module is ASCII-only. XXX could try wcstombs instead, if the file system encoding is the locale's encoding. */ else if (Py_FileSystemDefaultEncoding strcmp(encoding, Py_FileSystemDefaultEncoding) == 0 !PyThreadState_GET()-interp-codecs_initialized) return PyUnicode_AsASCIIString(unicode); I implemented the XXX could try wcstombs part, but in new functions: PyUnicode_EncodeFSDefault and PyUnicode_DecodeFSDefaultAndSize. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Graham Dumpleton graham.dumple...@gmail.com added the comment: Hmmm, I wander if that is related to the workaround I have added in mod_wsgi recently of: /* * Force loading of codecs into interpreter. This has to be * done as not otherwise done in sub interpreters and if not * done, code running in sub interpreters can fail on some * platforms if a unicode string is added in sys.path and an * import then done. */ item = PyCodec_Encoder(ascii); Py_XDECREF(item); This fixes problem some have seen where one gets: LookupError: no codec search functions registered: can't find encoding I haven't been able to reproduce the problem myself so no bug report ever lodged. Have been told it affects some other embedded systems as well which use sub interpreters but other systems don't employ the workaround that I am now using. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
STINNER Victor victor.stin...@haypocalc.com added the comment: The problem is the test on the python initialization: PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize check the global Py_FileSystemDefaultEncoding variable (use C functions if it is NULL). The problem is a little bit more complex. interp-codecs_initialized is set to 1 by _PyCodecRegistry_Init(), but the filesystem codec is not loaded yet. We need another variable: interp-fscodec_initialized. fscodec_subinterpreter.patch fixes this issue. Note: the patch replaces also PyErr_Print() by PyErr_PrintEx(0) to avoid a crash on error, if PyErr_PrintEx(0) is called before the sys module is initialized. For example, if PyModule_GetDict(_PyImport_FindBuiltin(builtins)) is NULL. Anyway, it's useless to set sys attributes on error, because sys is destroyed just after printing the error. -- Added file: http://bugs.python.org/file21778/fscodec_subinterpreter.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: After wrestling with startup issues on the OS X buildbots, here is a new patch, tested on several different UNIX platforms. -- Added file: http://bugs.python.org/file21765/embedtest2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Unless someone objects, I would like to commit that latest patch, since at least it enables the start of a regression suite for Python embedding. (it also allowed me to notice how crufty and incredibly obscure the getpath.c mechanisms are) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Nick Coghlan ncogh...@gmail.com added the comment: Sounds good. I still have that embedded pickle module issue to deal with, and this should let me actually add a test for it along with the fix. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Éric Araujo mer...@netwok.org added the comment: Note that I think it would be a perfectly reasonable feature request to better isolate and improve the compiler in packaging/distutils2. (This does not mean this issue should be delayed.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Éric Araujo mer...@netwok.org added the comment: Well, config._link() seems to do what is needed here. My point is that it’s easier to write a few lines of code directly using a compiler object (copying and simplifying code from try_run or _link) than go through the distutils command machinery. Both are doable, but I think the former is easier. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Ok, this is what I came up with to build an exe using distutils. At this point, the complication is downright silly and it doesn't even work under Windows, so we may prefer to go down the Makefile route instead. -- Added file: http://bugs.python.org/file21282/embedtest-distutils.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Roundup Robot devnull@devnull added the comment: New changeset a791dd7d51f3 by Antoine Pitrou in branch '3.2': Issue #10914: fix bogus memory management in Modules/getpath.c, leading to a possible crash when calling Py_SetPath() http://hg.python.org/cpython/rev/a791dd7d51f3 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Ouch, sorry, I got my commit message wrong. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Éric Araujo mer...@netwok.org added the comment: distutils sure knows how to build .o or .so files, but I don’t know about standalone executables (because I don’t know how the .o end up making an executable). If you want to try to do it, I would advise you not to use the config command but rather a compiler object directly. distutils.ccompiler.new_compiler() should give you an instance of a subclass of distutils.ccompiler.CCompiler suitable for your system, and then you can use help or the source to find what methods to call. -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: distutils sure knows how to build .o or .so files, but I don’t know about standalone executables (because I don’t know how the .o end up making an executable). If you want to try to do it, I would advise you not to use the config command but rather a compiler object directly. distutils.ccompiler.new_compiler() should give you an instance of a subclass of distutils.ccompiler.CCompiler suitable for your system, and then you can use help or the source to find what methods to call. Well, config._link() seems to do what is needed here. If you think it is bad to rely on it, perhaps we should inline its code somehow in the test module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Nick Coghlan ncogh...@gmail.com added the comment: I like the idea of adding these tests as well. Probably worth bringing up on python-dev - folks like Tarek should definitely be able to help with alternative ways of building a test application for this. There is also always the possibility of writing it in Python and using ctypes to get at the necessary C API calls to create subinterpreters. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: There is also always the possibility of writing it in Python and using ctypes to get at the necessary C API calls to create subinterpreters. Well, I don't think calling Py_Initialize, Py_Finalize and Py_NewInterpreter from pure Python code is a very good idea. Tarek, can you help? We'd like to use distutils to compile an executable embedding the interpreter. -- nosy: +tarek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: +1 for this kind of tests. But I would not have their source in the official Modules directory. I expect that there will be several tests of this kind, each one with different modules to import, functions to run, global settings to change. IMO the C code should be somewhere in the test directory; later we could use templates to generate source code for similar tests. Is it possible to use distutils to compile this test, and remove it from the Makefile? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Benjamin Peterson benja...@python.org added the comment: There's already precedent for test modules in Modules/ what with xxmodule.c and _testcapimodule.c. -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Antoine Pitrou pit...@free.fr added the comment: Is it possible to use distutils to compile this test, and remove it from the Makefile? I'm not aware that distutils is able to compile applications rather than extension modules, but that would certainly be better than a rule in the Makefile, yes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: In distutils.command.config, config().try_run(body) is probably what we need. Now, I don't know how to use it... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
New submission from Antoine Pitrou pit...@free.fr: Here is a prototype test for embedding and sub-interpreters. -- components: Interpreter Core, Tests files: embedtest.patch keywords: patch messages: 126328 nosy: christian.heimes, grahamd, haypo, loewis, pitrou priority: normal severity: normal status: open title: Python sub-interpreter test versions: Python 3.2, Python 3.3 Added file: http://bugs.python.org/file20414/embedtest.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10914] Python sub-interpreter test
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +amaury.forgeotdarc, ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10914 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
