[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Do you have access to Windows? I believe it doesn't have dup(). :-( __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Guido van Rossum wrote: Guido van Rossum added the comment: Do you have access to Windows? I believe it doesn't have dup(). :-( I've an old laptop with Win2k at my disposal but it has no VS yet. Can you point me to a set of instruction how to install VS 2003? I've just VS .NET 2005 available. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Do you have access to Windows? I believe it doesn't have dup(). :-( I've an old laptop with Win2k at my disposal but it has no VS yet. Can you point me to a set of instruction how to install VS 2003? I've just VS .NET 2005 available. Sorry, I'm as green as you when it comes to these. :-( But I believe I was mistaken about dup() not existing on Windows; dup() can't be used to duplicate sockets, but that's irrelevant here. So the dup()-based solution is fine. Alas, my family won't let me use the computer for more than a minute at a time today, so I won't be able to review any code... :-) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Changes by Guido van Rossum: -- resolution: accepted - status: closed - open __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: OK, checked in. You might want to compare what I checked in to your patch; I did a few style cleanups. I also moved the lseek() call into import.c, where it seems more appropriate. Committed revision 58587. -- resolution: - accepted status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Guido van Rossum wrote: You might want to compare what I checked in to your patch; I did a few style cleanups. I also moved the lseek() call into import.c, where it seems more appropriate. Ah I see that you prefer to keep assignment and check against NULL/-1 on two separate lines. I had the lseek() in PyTokenizer_FindEncoding() because I prefer functions that restore their environment. I find it less surprising when it restores the position of the file descriptor. By the way I got Windows, VS 2003 and several SDKs installed in VMWare today. It's annoying and it takes hours. Most unit tests are passing. http://wiki.python.org/moin/Building_Python_with_the_free_MS_C_Toolkit __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Thanks for persevering!!! The dangers of switching between fileno(fp) and fp are actually well documented in the C and/or POSIX standards. The problem is caused in PyFile_FromFileEx() -- it creates a Python file object from the file descriptor. The fix actually only works because we're not using the FILE struct once PyTokenizer_FindEncoding() is called. I think it would be better to move the lseek() into call_find_module() so the FILE abstraction is not broken by PyTokenizer_FindEncoding(). I think there's still a bug or two lurking in this area: first, each time you call imp.find_module() you leak a FILE object; second, the encoding allocated in PyTokenizer_FindEncoding() is leaked. You're right that a lot of this could be avoided if we used file descriptors consistently. It seems find_module() itself doesn't read the file; it just needs to know that it's possible to open the file. Rewriting everywhere that uses PyFile_FromFile[Ex] to use file descriptors doesn't seem too hard; there are only a few places. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Sorry, I was wrong about the encoding leak; you fixed that. The FILE/fd issue is real though. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Guido van Rossum wrote: You're right that a lot of this could be avoided if we used file descriptors consistently. It seems find_module() itself doesn't read the file; it just needs to know that it's possible to open the file. Rewriting everywhere that uses PyFile_FromFile[Ex] to use file descriptors doesn't seem too hard; there are only a few places. If I understand you right you want to alter the interface of PyFile_FromFile and PyFile_FromFileEx from PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE*), int buffering, char *encoding, char *newline) to PyFile_FromFileEx(int fd, char *name, char *mode, int (*close)(FILE*), int buffering, char *encoding, char *newline) ? I could write a patch and remove int (*close)(FILE*), too. It's no longer used by the functions. Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Yes, I think you got it. Guido, Brett, how much of the code should I change? import.c is mainly using file pointers. Should I replace all operations on FILE with operations on a file descriptor? Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Guido, Brett, how much of the code should I change? import.c is mainly using file pointers. Should I replace all operations on FILE with operations on a file descriptor? I think find_module() should return a file descriptor (fd), not a FILE*, but most of the rest of the code should call fdopen() on that fd. Only call_find_module() should use the fd to turn it into a Python file object. Then the amount of change should be pretty minimal. I recommend changing the name of the API used to turn a fd into file object while we're at it; that's one of the few guidelines we have for C API changes. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
Guido van Rossum wrote:
I think find_module() should return a file descriptor (fd), not a
FILE*, but most of the rest of the code should call fdopen() on that
fd. Only call_find_module() should use the fd to turn it into a Python
file object. Then the amount of change should be pretty minimal.
K, I understand.
I recommend changing the name of the API used to turn a fd into file
object while we're at it; that's one of the few guidelines we have for
C API changes.
Is PyFile_FromFd and PyFile_FromFdEx fine with you or do you prefer a
different name like PyFile_FromFD and PyFile_FromFDEx or
PyFile_FromFileDescriptor?
I've another question. The os.tmpfile method is using tmpfile() which
returns a file pointer. Do I have to dup its file number and explictely
close the file pointer as shown below or is a simple fileno() enough? I
haven't found sample code how to use a file descriptor after the file
pointer is closed.
static PyObject *
posix_tmpfile(PyObject *self, PyObject *noargs)
{
FILE *fp;
int fd;
fp = tmpfile();
if (fp == NULL)
return posix_error();
fd = dup(fileno(fp));
if (fd == -1)
return posix_error();
fclose(fp);
return PyFile_FromFD(fd, tmpfile, w+b);
}
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Is PyFile_FromFd and PyFile_FromFdEx fine with you or do you prefer a different name like PyFile_FromFD and PyFile_FromFDEx or PyFile_FromFileDescriptor? Let's have a single function PyFile_FromFd(fd, name, mode, buffering, encoding, newline). I've another question. The os.tmpfile method is using tmpfile() which returns a file pointer. Do I have to dup its file number and explictely close the file pointer as shown below or is a simple fileno() enough? You should dup it; if you don't dup it, the fd will be closed when you call fclose(), rendering it useless; or when you don't call fclose(), you leak a FILE struct each time. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
I think find_module() should return a file descriptor (fd), not a
FILE*, but most of the rest of the code should call fdopen() on that
fd. Only call_find_module() should use the fd to turn it into a Python
file object. Then the amount of change should be pretty minimal.
I'd have to touch most functions in import.c and related files to change
find_module() to use a file descriptor. It's a PITA and I don't think
it's worse the effort for now.
The new patch adds PyFile_FromFd and removes the other PyFile_FromFile*
functions. It also changes some methods to use a file descriptor and
some documentation. Two minor changes aren't related to the bug but they
nagged me.
Added file: http://bugs.python.org/file8585/py3k_filefromfd.patch
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__Index: Python/pythonrun.c
===
--- Python/pythonrun.c (revision 58574)
+++ Python/pythonrun.c (working copy)
@@ -719,7 +719,7 @@
}
/* Set sys.stdin */
- if (!(std = PyFile_FromFileEx(stdin, stdin, r, fclose, -1,
+ if (!(std = PyFile_FromFd(STDIN_FILENO, stdin, r, -1,
NULL, \n))) {
goto error;
}
@@ -728,7 +728,7 @@
Py_DECREF(std);
/* Set sys.stdout */
- if (!(std = PyFile_FromFileEx(stdout, stdout, w, fclose, -1,
+ if (!(std = PyFile_FromFd(STDOUT_FILENO, stdout, w, -1,
NULL, \n))) {
goto error;
}
@@ -737,7 +737,7 @@
Py_DECREF(std);
/* Set sys.stderr */
- if (!(std = PyFile_FromFileEx(stderr, stderr, w, fclose, -1,
+ if (!(std = PyFile_FromFd(STDERR_FILENO, stderr, w, -1,
NULL, \n))) {
goto error;
}
Index: Python/import.c
===
--- Python/import.c (revision 58574)
+++ Python/import.c (working copy)
@@ -92,7 +92,7 @@
extern struct _inittab _PyImport_Inittab[];
/* Method from Parser/tokenizer.c */
-extern char * PyTokenizer_FindEncoding(FILE *fp);
+extern char * PyTokenizer_FindEncoding(int);
struct _inittab *PyImport_Inittab = _PyImport_Inittab;
@@ -2561,6 +2561,7 @@
struct filedescr *fdp;
char pathname[MAXPATHLEN+1];
FILE *fp = NULL;
+ int fd = -1;
char *found_encoding = NULL;
char *encoding = NULL;
@@ -2571,17 +2572,24 @@
if (fdp == NULL)
return NULL;
if (fp != NULL) {
+ fd = fileno(fp);
+ if (fd != -1)
+ fd = dup(fd);
+ fclose(fp);
+ fp = NULL;
+ }
+ if (fd != -1) {
if (strchr(fdp-mode, 'b') == NULL) {
/* PyTokenizer_FindEncoding() returns PyMem_MALLOC'ed
memory. */
- found_encoding = PyTokenizer_FindEncoding(fp);
+ found_encoding = PyTokenizer_FindEncoding(fd);
encoding = (found_encoding != NULL) ? found_encoding :
(char*)PyUnicode_GetDefaultEncoding();
}
- fob = PyFile_FromFileEx(fp, pathname, fdp-mode, fclose, -1,
+ fob = PyFile_FromFd(fd, pathname, fdp-mode, -1,
(char*)encoding, NULL);
if (fob == NULL) {
- fclose(fp);
+ close(fd);
PyMem_FREE(found_encoding);
return NULL;
}
Index: Include/fileobject.h
===
--- Include/fileobject.h (revision 58574)
+++ Include/fileobject.h (working copy)
@@ -8,10 +8,7 @@
#define PY_STDIOTEXTMODE b
-PyAPI_FUNC(PyObject *) PyFile_FromFile(FILE *, char *, char *, int (*)(FILE*));
-PyAPI_FUNC(PyObject *) PyFile_FromFileEx(FILE *, char *, char *,
- int (*)(FILE *), int, char *,
- char *);
+PyAPI_FUNC(PyObject *) PyFile_FromFd(int, char *, char *, int, char *, char *);
PyAPI_FUNC(PyObject *) PyFile_GetLine(PyObject *, int);
PyAPI_FUNC(int) PyFile_WriteObject(PyObject *, PyObject *, int);
PyAPI_FUNC(int) PyFile_WriteString(const char *, PyObject *);
Index: Objects/fileobject.c
===
--- Objects/fileobject.c (revision 58574)
+++ Objects/fileobject.c (working copy)
@@ -26,21 +26,15 @@
/* External C interface */
PyObject *
-PyFile_FromFile(FILE *fp, char *name, char *mode, int (*close)(FILE *))
+PyFile_FromFd(int fd, char *name, char *mode, int buffering, char *encoding,
+ char *newline)
{
- return PyFile_FromFileEx(fp, name, mode, close, -1, NULL, NULL);
-}
-
-PyObject *
-PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE *),
- int buffering, char *encoding, char *newline)
-{
PyObject *io, *stream, *nameobj=NULL;
io = PyImport_ImportModule(io);
if (io == NULL)
return NULL;
- stream = PyObject_CallMethod(io, open, isiss, fileno(fp), mode,
+ stream = PyObject_CallMethod(io, open, isiss, fd, mode,
buffering, encoding, newline);
Py_DECREF(io);
if (stream == NULL)
Index: Objects/bytesobject.c
===
--- Objects/bytesobject.c (revision 58574)
+++ Objects/bytesobject.c (working copy)
@@ -1214,7 +1214,7 @@
Py_ssize_t
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: I will take this. -- assignee: brett.cannon - gvanrossum __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: Committed revision 58553 (with minor tweaks only). Thanks! -- resolution: - accepted status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: runpy is failing because pkgutil is failing because it is giving compile() part of a source file instead of the entire thing:: Traceback (most recent call last): File /Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py, line 97, in _run_module_as_main loader, code, fname = _get_module_details(mod_name) File /Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py, line 82, in _get_module_details code = loader.get_code(mod_name) File /Users/drifty/Dev/python/3.x/pristine/Lib/pkgutil.py, line 275, in get_code self.code = compile(source, self.filename, 'exec') File /Users/drifty/Dev/python/3.x/pristine/Lib/tokenize.py, line 2 UR''': single3prog, 'UR': double3prog, ^ IndentationError: unexpected indent That bad line is the first line in the 'source' variable being passed to compile(). So somewhere the beginning of the source file is being chopped up. On 10/19/07, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: I don't have a Mac at my disposal any more. :( Can you attach the output of the failing tests to the bug report? Maybe I can be of assistance. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
Brett Cannon wrote:
Brett Cannon added the comment:
It looks like the file object returned by imp.find_module() has its read
position WAY too far forward (at least on OS X).
That's strange. It should never read more than 2 lines of a file. I
don't understand how it could happen.
char *
PyTokenizer_FindEncoding(FILE *fp) {
struct tok_state *tok;
char *p_start=NULL, *p_end=NULL;
if ((tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL)) == NULL) {
rewind(fp);
return NULL;
}
while(((tok-lineno = 2) (tok-done == E_OK))) {
PyTokenizer_Get(tok, p_start, p_end);
}
rewind(fp);
return tok-encoding;
}
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: runpy is failing because pkgutil is failing because it is giving compile() part of a source file instead of the entire thing:: Traceback (most recent call last): File /Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py, line 97, in _run_module_as_main loader, code, fname = _get_module_details(mod_name) File /Users/drifty/Dev/python/3.x/pristine/Lib/runpy.py, line 82, in _get_module_details code = loader.get_code(mod_name) File /Users/drifty/Dev/python/3.x/pristine/Lib/pkgutil.py, line 275, in get_code self.code = compile(source, self.filename, 'exec') File /Users/drifty/Dev/python/3.x/pristine/Lib/tokenize.py, line 2 UR''': single3prog, 'UR': double3prog, ^ IndentationError: unexpected indent That bad line is the first line in the 'source' variable being passed to compile(). So somewhere the beginning of the source file is being chopped up. Could you please comment out the PyTokenizer_FindEncoding(fp) call in Python/import.c to check if it is related to it? That's the only change (I can think of) that may be related to the problem. I always rewind the fp in the function but it may not work on Mac. Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: It looks like the file object returned by imp.find_module() has its read position WAY too far forward (at least on OS X). Re-opening. -- status: closed - open __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
It's unrelated to the problem but Parser/tokenizer.c:1619 has a minor bug.
while(((tok-lineno = 2) (tok-done == E_OK))) {
PyTokenizer_Get(tok, p_start, p_end);
}
(tok-lineno 2) is sufficient. See line 516
Christian
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: I don't have a Mac at my disposal any more. :( Can you attach the output of the failing tests to the bug report? Maybe I can be of assistance. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: Yeah, I just noticed this as well. is it definitely this change? On 10/19/07, Guido van Rossum [EMAIL PROTECTED] wrote: Guido van Rossum added the comment: FWIW, on Mac I get six failures (all these pass on Linux): test_cmd_line test_inspect test_modulefinder test_pyclbr test_quopri test_runpy __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: OK, for some reason, when PyTokenizer_FindEncoding() is called and the resulting file is big enough, it starts off with a seek position (according to TextIOWrapper.tell()) of 4096. That happens to be exactly half the size of the buffer used by io. But I am not sure what the magical size is as creating a file of 4095, 4096, and 4097 bytes does not trigger this bug. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: Nope, didn't do it. I also checked using gdb a few minutes ago and ftell() was reporting a position of 0 all they way back to PyObject_MethodCall(). __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: 58557 as the fix. Yes, it was stupid on OS X's part and I was lucky to just think of the possible solution. Basically OS X does not like it when you do stuff with a file pointer but then poke around with the file descriptor. That means that when PyTokenizer_FindEncoding() seeked on the file pointer it was not being picked up by the the file descriptor that the file pointer represented. This suggests that perhaps we should standardize on file pointers or file descriptors in Python to prevent something like this from happening again. -- assignee: gvanrossum - brett.cannon status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: OK, I think I might have a solution, and it is really stupid. I will run the unit test suite to see if it really fixes everything. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Brett Cannon wrote: Brett Cannon added the comment: OK, for some reason, when PyTokenizer_FindEncoding() is called and the resulting file is big enough, it starts off with a seek position (according to TextIOWrapper.tell()) of 4096. That happens to be exactly half the size of the buffer used by io. But I am not sure what the magical size is as creating a file of 4095, 4096, and 4097 bytes does not trigger this bug. Maybe rewind() doesn't do what is should do? Could you replace the rewind() calls with fseek(fd, 0, 0)? w/o the rewind() calls in PyTokenizer_FindEncoding I'm getting an error in my new test: Traceback (most recent call last): File Lib/test/test_imp.py, line 66, in module test_main() File Lib/test/test_imp.py, line 62, in test_main ImportTests, File /home/heimes/dev/python/py3k/Lib/test/test_support.py, line 541, in run_unittest _run_suite(suite) File /home/heimes/dev/python/py3k/Lib/test/test_support.py, line 524, in _run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File Lib/test/test_imp.py, line 50, in test_issue1267 self.assertEqual(fd.tell(), 0) AssertionError: 4096 != 0 The number is looking familiar, isn't it? :) Is it the default buffer size on Unix? Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: I've made a short unit tests which tests a large file with and w/o -*- coding: -*-. It passes on Linux. Added file: http://bugs.python.org/file8575/py3k_test_issue1267.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __Index: Lib/test/test_imp.py === --- Lib/test/test_imp.py (revision 58554) +++ Lib/test/test_imp.py (working copy) @@ -44,6 +44,18 @@ fd = imp.find_module(heapq)[0] self.assertEqual(fd.encoding, iso-8859-1) +def test_issue1267(self): +fd, filename, info = imp.find_module(pydoc) +self.assertEqual(fd.encoding, iso-8859-1) +self.assertEqual(fd.tell(), 0) +self.assertEqual(fd.readline(), '#!/usr/bin/env python\n') + +fd, filename, info = imp.find_module(tokenize) +self.assertEqual(fd.encoding, utf-8) +self.assertEqual(fd.tell(), 0) +self.assertEqual(fd.readline(), 'Tokenization help for Python programs.\n') + + def test_main(): test_support.run_unittest( LockTests, ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: OK, I think I might have a solution, and it is really stupid. I will run the unit test suite to see if it really fixes everything. Here's keeping my fingers crossed. I do note that the new code still leaks the encoding string which is malloc'ed in PyTokenizer_FindEncoding but never freed by call_find_module. BTW, the test_inspect failure I saw was shallow; I'd renamed the parent directory and the failure was due to the wrong filename being baked into the .pyc file. The other 5 failures are all related to the issue you are chasing here. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: I checked in your `` 2`` change and plugged a memory leak since you were not freeing the struct tok_state. Checked in r58555. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: On 10/19/07, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: Brett Cannon wrote: This suggests that perhaps we should standardize on file pointers or file descriptors in Python to prevent something like this from happening again. rewind() it used couple of times in the Python code. Have you checked if the other calls are safe? No. I am still rather frustrated that was the problem. The only reason I feel safe with this solution is that the file pointer is not directly used except by _fileio to get the file descriptor. I would not trust this if the file pointer and file descriptor had intertwined uses. But I would trust rewind() if the file pointer is used the entire time. Thx for finding the bug :) Great work! Thanks. I just wish this whole ordeal had not been necessary (filed a bug report with Apple in hopes that this can be prevented for someone else). I can see why some people prefer to hack on PyPy, IronPython, or Jython instead of dealing with the joys of C. =) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment: FWIW, on Mac I get six failures (all these pass on Linux): test_cmd_line test_inspect test_modulefinder test_pyclbr test_quopri test_runpy __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Brett Cannon wrote: This suggests that perhaps we should standardize on file pointers or file descriptors in Python to prevent something like this from happening again. rewind() it used couple of times in the Python code. Have you checked if the other calls are safe? Thx for finding the bug :) Great work! Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: PyTokenizer_FindEncoding() is the culprit. If I comment it out in Python/import.c:call_find_module(), and thus just use the system encoding, runpy works again. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment: Brett Cannon wrote: Thanks. I just wish this whole ordeal had not been necessary (filed a bug report with Apple in hopes that this can be prevented for someone else). I can see why some people prefer to hack on PyPy, IronPython, or Jython instead of dealing with the joys of C. =) The bug is rather strange. I would have imagined that fseek() and rewind() are using the file descriptor internally. It's more than weird that an operation on the file handler doesn't affect the file descriptor. I wonder how they were able to manage it ... IronPython might be fun but I've hacked PythonDotNet. You get the worst from the C, .NET/C# and Mono together. Debugging is fun when you have to do multiple turns with two debuggers and one of the debuggers is partly broken. :[ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Brett Cannon added the comment: Thanks, Guido. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
Christian Heimes wrote:
* removed unused import of open in initstdio()
* fixed infinite loop in PyTokenizer_FindEncoding() by checking
tok-done == E_OK
I found another bug in Python/import.c:call_find_method. The function
mustn't set an encoding of ftp-mode contains 'b' for binary.
if (strchr(fdp-mode, 'b') == NULL) {
/* Python text file, get encoding from tokenizer */
encoding = PyTokenizer_FindEncoding(fp);
encoding = (encoding != NULL) ? encoding :
PyUnicode_GetDefaultEncoding();
}
Christian
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment:
Does this mean I should hold off reviewing the patch?
On 10/16/07, Christian Heimes [EMAIL PROTECTED] wrote:
Christian Heimes added the comment:
Christian Heimes wrote:
* removed unused import of open in initstdio()
* fixed infinite loop in PyTokenizer_FindEncoding() by checking
tok-done == E_OK
I found another bug in Python/import.c:call_find_method. The function
mustn't set an encoding of ftp-mode contains 'b' for binary.
if (strchr(fdp-mode, 'b') == NULL) {
/* Python text file, get encoding from tokenizer */
encoding = PyTokenizer_FindEncoding(fp);
encoding = (encoding != NULL) ? encoding :
PyUnicode_GetDefaultEncoding();
}
Christian
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
Update since last patch
* removed unnecessary const from const char*
PyTokenizer_FindEncoding(FILE *fp)
* Fixed bug in find_module whith binary files
Please review the patch.
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__Index: Python/pythonrun.c
===
--- Python/pythonrun.c (revision 58495)
+++ Python/pythonrun.c (working copy)
@@ -51,6 +51,7 @@
/* Forward */
static void initmain(void);
static void initsite(void);
+static int initstdio(void);
static PyObject *run_mod(mod_ty, const char *, PyObject *, PyObject *,
PyCompilerFlags *, PyArena *);
static PyObject *run_pyc_file(FILE *, const char *, PyObject *, PyObject *,
@@ -241,6 +242,9 @@
initsigs(); /* Signal handling stuff, including initintr() */
initmain(); /* Module __main__ */
+ if (initstdio() 0)
+ Py_FatalError(
+ Py_Initialize: can't initialize sys standard streams);
if (!Py_NoSiteFlag)
initsite(); /* Module site */
@@ -676,6 +680,81 @@
}
}
+/* Initialize sys.stdin, stdout, stderr and __builtin__.open */
+static int
+initstdio(void)
+{
+ PyObject *iomod=NULL, *wrapper;
+ PyObject *bimod=NULL;
+ PyObject *m;
+ PyObject *std=NULL;
+ int status = 0;
+
+ /* Hack to avoid a nasty recursion issue when Python is invoked
+ in verbose mode: pre-import the Latin-1 and UTF-8 codecs */
+ if ((m = PyImport_ImportModule(encodings.utf_8)) == NULL) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(m = PyImport_ImportModule(encodings.latin_1))) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(bimod = PyImport_ImportModule(__builtin__))) {
+ goto error;
+ }
+
+ if (!(iomod = PyImport_ImportModule(io))) {
+ goto error;
+ }
+ if (!(wrapper = PyObject_GetAttrString(iomod, OpenWrapper))) {
+ goto error;
+ }
+
+ /* Set __builtin__.open */
+ if (PyObject_SetAttrString(bimod, open, wrapper) == -1) {
+ goto error;
+ }
+
+ /* Set sys.stdin */
+ if (!(std = PyFile_FromFileEx(stdin, stdin, r, fclose, -1,
+ NULL, \n))) {
+ goto error;
+ }
+ PySys_SetObject(__stdin__, std);
+ PySys_SetObject(stdin, std);
+ Py_DECREF(std);
+
+ /* Set sys.stdout */
+ if (!(std = PyFile_FromFileEx(stdout, stdout, w, fclose, -1,
+ NULL, \n))) {
+goto error;
+}
+ PySys_SetObject(__stdout__, std);
+ PySys_SetObject(stdout, std);
+ Py_DECREF(std);
+
+ /* Set sys.stderr */
+ if (!(std = PyFile_FromFileEx(stderr, stderr, w, fclose, -1,
+ NULL, \n))) {
+goto error;
+}
+PySys_SetObject(__stderr__, std);
+ PySys_SetObject(stderr, std);
+ Py_DECREF(std);
+
+if (0) {
+ error:
+status = -1;
+}
+
+ Py_XDECREF(bimod);
+ Py_XDECREF(iomod);
+ return status;
+}
+
/* Parse input from a file and execute it */
int
@@ -1146,10 +1225,10 @@
int err = 0;
PyObject *f = PySys_GetObject(stderr);
Py_INCREF(value);
- if (f == NULL)
+ if (f == NULL) {
_PyObject_Dump(value);
- if (f == NULL)
fprintf(stderr, lost sys.stderr\n);
+ }
else {
fflush(stdout);
if (tb tb != Py_None)
@@ -1589,6 +1668,9 @@
Py_FatalError(const char *msg)
{
fprintf(stderr, Fatal Python error: %s\n, msg);
+ if (PyErr_Occurred()) {
+ PyErr_Print();
+ }
#ifdef MS_WINDOWS
OutputDebugString(Fatal Python error: );
OutputDebugString(msg);
Index: Python/import.c
===
--- Python/import.c (revision 58495)
+++ Python/import.c (working copy)
@@ -91,6 +91,9 @@
/* This table is defined in config.c: */
extern struct _inittab _PyImport_Inittab[];
+/* Method from Parser/tokenizer.c */
+extern char * PyTokenizer_FindEncoding(FILE *fp);
+
struct _inittab *PyImport_Inittab = _PyImport_Inittab;
/* these tables define the module suffixes that Python recognizes */
@@ -2558,6 +2561,7 @@
struct filedescr *fdp;
char pathname[MAXPATHLEN+1];
FILE *fp = NULL;
+ char *encoding = NULL;
pathname[0] = '\0';
if (path == Py_None)
@@ -2566,7 +2570,14 @@
if (fdp == NULL)
return NULL;
if (fp != NULL) {
- fob = PyFile_FromFile(fp, pathname, fdp-mode, fclose);
+ if (strchr(fdp-mode, 'b') == NULL) {
+ /* Python text file, get encoding from tokenizer */
+ encoding = PyTokenizer_FindEncoding(fp);
+ encoding = (encoding != NULL) ? encoding :
+ (char*)PyUnicode_GetDefaultEncoding();
+ }
+ fob = PyFile_FromFileEx(fp, pathname, fdp-mode, fclose, -1,
+ (char*)encoding, NULL);
if (fob == NULL) {
fclose(fp);
return NULL;
Index: Include/fileobject.h
===
--- Include/fileobject.h (revision 58495)
+++ Include/fileobject.h (working copy)
@@ -9,6 +9,9 @@
#define PY_STDIOTEXTMODE b
PyAPI_FUNC(PyObject *) PyFile_FromFile(FILE *, char *, char *, int (*)(FILE*));
+PyAPI_FUNC(PyObject *) PyFile_FromFileEx(FILE *, char *, char *,
+ int (*)(FILE *), int, char *,
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
I've some code around which sets sys.stdin, out and err in C code. The
code is far from perfect and I haven't checked it for reference leaks
yet. I like to get your comment on the style and error catching.
--
nosy: +tiran
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__Index: Python/pythonrun.c
===
--- Python/pythonrun.c (revision 58477)
+++ Python/pythonrun.c (working copy)
@@ -51,6 +51,7 @@
/* Forward */
static void initmain(void);
static void initsite(void);
+static int initsysstd(void);
static PyObject *run_mod(mod_ty, const char *, PyObject *, PyObject *,
PyCompilerFlags *, PyArena *);
static PyObject *run_pyc_file(FILE *, const char *, PyObject *, PyObject *,
@@ -241,8 +242,11 @@
initsigs(); /* Signal handling stuff, including initintr() */
initmain(); /* Module __main__ */
+ if (initsysstd() 0)
+ Py_FatalError(Py_Initialize: can't initialize sys std streams);
if (!Py_NoSiteFlag)
initsite(); /* Module site */
+
/* auto-thread-state API, if available */
#ifdef WITH_THREAD
@@ -676,6 +680,93 @@
}
}
+/* Initialize sys.stdin, stdout and stderr */
+static int
+initsysstd(void)
+{
+ PyObject *io=NULL, *iod, *open;
+ PyObject *bi=NULL, *bid;
+ PyObject *m, *wrapper;
+ PyObject *std, *args=NULL, *kwargs=NULL;
+
+ /* Hack to avoid a nasty recursion issue when Python is invoked
+ in verbose mode: pre-import the Latin-1 and UTF-8 codecs */
+ if ((m = PyImport_ImportModule(encodings.utf_8)) == NULL)
+ return -1;
+ Py_DECREF(m);
+
+ if ((m = PyImport_ImportModule(encodings.latin_1)) == NULL)
+ return -1;
+ Py_DECREF(m);
+
+ if ((bi = PyImport_ImportModule(__builtin__)) == NULL)
+ goto error;
+ if ((bid = PyModule_GetDict(bi)) == NULL)
+ goto error;
+
+ if ((io = PyImport_ImportModule(io)) == NULL)
+ goto error;
+ if ((iod = PyModule_GetDict(io)) == NULL)
+ goto error;
+
+ if ((wrapper = PyDict_GetItemString(iod, OpenWrapper)) == NULL)
+ goto error;
+ Py_INCREF(wrapper);
+ if (PyDict_SetItemString(bid, open, wrapper) == -1) {
+ Py_DECREF(wrapper); /* ??? */
+ goto error;
+ }
+
+ if ((open = PyDict_GetItemString(iod, open)) == NULL)
+ goto error;
+
+ if ((kwargs = PyDict_New()) == NULL)
+ goto error;
+ if (PyDict_SetItemString(kwargs, newline, PyString_FromString(\n))
+ == -1)
+ goto error;
+
+ /* stdin */
+ if ((args = Py_BuildValue((is), 0, r)) == NULL)
+ goto error;
+ if ((std = PyObject_Call(open, args, kwargs)) == NULL)
+ goto error;
+ PySys_SetObject(__stdin__, std);
+ PySys_SetObject(stdin, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ /* stdout */
+ if ((args = Py_BuildValue((is), 1, w)) == NULL)
+ goto error;
+ if ((std = PyObject_Call(open, args, kwargs)) == NULL)
+ goto error;
+ PySys_SetObject(__stdout__, std);
+ PySys_SetObject(stdout, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ /* stderr */
+ if ((args = Py_BuildValue((is), 2, w)) == NULL)
+ goto error;
+ if ((std = PyObject_Call(open, args, kwargs)) == NULL)
+ goto error;
+ PySys_SetObject(__stderr__, std);
+ PySys_SetObject(stderr, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ return 0;
+
+ error:
+ Py_XDECREF(bi);
+ Py_XDECREF(io);
+ Py_XDECREF(kwargs);
+ Py_XDECREF(args);
+ return -1;
+
+}
+
/* Parse input from a file and execute it */
int
Index: Lib/site.py
===
--- Lib/site.py (revision 58477)
+++ Lib/site.py (working copy)
@@ -433,7 +433,7 @@
sethelper()
aliasmbcs()
setencoding()
-installnewio()
+#installnewio()
execsitecustomize()
# Remove sys.setdefaultencoding() so that users cannot change the
# encoding after initialization. The test for presence is needed when
Index: Lib/io.py
===
--- Lib/io.py (revision 58477)
+++ Lib/io.py (working copy)
@@ -178,6 +178,16 @@
return text
+class OpenWrapper:
+Wrapper for __builtin__.open
+
+Trick so that open won't become a bound method when stored
+as a class variable (as dumbdbm does)
+
+def __new__(cls, *args, **kwargs):
+return open(*args, **kwargs)
+
+
class UnsupportedOperation(ValueError, IOError):
pass
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Changes by Brett Cannon: -- assignee: - brett.cannon keywords: +patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
I've carefully checked and tested the initstdio() method. I'm sure that
I've catched every edged case. The unit tests pass w/o complains.
I've also added a PyErr_Display() call to Py_FatalError(). It's still
hard to understand an error in io.py but at least the dependency on
site.py is removed.
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__Index: Python/pythonrun.c
===
--- Python/pythonrun.c (revision 58477)
+++ Python/pythonrun.c (working copy)
@@ -51,6 +51,7 @@
/* Forward */
static void initmain(void);
static void initsite(void);
+static int initstdio(void);
static PyObject *run_mod(mod_ty, const char *, PyObject *, PyObject *,
PyCompilerFlags *, PyArena *);
static PyObject *run_pyc_file(FILE *, const char *, PyObject *, PyObject *,
@@ -241,8 +242,12 @@
initsigs(); /* Signal handling stuff, including initintr() */
initmain(); /* Module __main__ */
+ if (initstdio() 0)
+ Py_FatalError(Py_Initialize: can't initialize sys standard
+streams);
if (!Py_NoSiteFlag)
initsite(); /* Module site */
+
/* auto-thread-state API, if available */
#ifdef WITH_THREAD
@@ -676,6 +681,103 @@
}
}
+/* Initialize sys.stdin, stdout, stderr and __builtins__.open */
+static int
+initstdio(void)
+{
+ PyObject *iomod=NULL, *open, *wrapper;
+ PyObject *bimod=NULL;
+ PyObject *m;
+ PyObject *std=NULL, *args=NULL, *kwargs=NULL;
+ int status = 0;
+
+ /* Hack to avoid a nasty recursion issue when Python is invoked
+ in verbose mode: pre-import the Latin-1 and UTF-8 codecs */
+ if ((m = PyImport_ImportModule(encodings.utf_8)) == NULL) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(m = PyImport_ImportModule(encodings.latin_1))) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(bimod = PyImport_ImportModule(__builtin__))) {
+ goto error;
+ }
+
+ if (!(iomod = PyImport_ImportModule(io))) {
+ goto error;
+ }
+ if (!(wrapper = PyObject_GetAttrString(iomod, OpenWrapper))) {
+ goto error;
+ }
+ if (!(open = PyObject_GetAttrString(iomod, open))) {
+ goto error;
+ }
+
+ if (PyObject_SetAttrString(bimod, open, wrapper) == -1) {
+ goto error;
+ }
+
+ /* Add __builtin__.open */
+ if (!(kwargs = PyDict_New())) {
+ goto error;
+ }
+ if (PyDict_SetItemString(kwargs, newline, PyString_FromString(\n))
+ == -1) {
+ goto error;
+ }
+
+ /* Set sys.stdin */
+ if (!(args = Py_BuildValue((is), 0, r))) {
+ goto error;
+ }
+ if (!(std = PyObject_Call(open, args, kwargs))) {
+ goto error;
+ }
+ PySys_SetObject(__stdin__, std);
+ PySys_SetObject(stdin, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ /* Set sys.stdout */
+ if (!(args = Py_BuildValue((is), 1, w))) {
+ goto error;
+ }
+ if (!(std = PyObject_Call(open, args, kwargs))) {
+ goto error;
+ }
+ PySys_SetObject(__stdout__, std);
+ PySys_SetObject(stdout, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ /* Set sys.stderr */
+ if (!(args = Py_BuildValue((is), 2, w))) {
+ goto error;
+ }
+ if (!(std = PyObject_Call(open, args, kwargs))) {
+ goto error;
+ }
+ PySys_SetObject(__stderr__, std);
+ PySys_SetObject(stderr, std);
+ Py_DECREF(std);
+ Py_DECREF(args);
+
+ goto finally;
+
+error:
+ status = -1;
+ Py_XDECREF(args);
+finally:
+ Py_XDECREF(kwargs);
+ Py_XDECREF(bimod);
+ Py_XDECREF(iomod);
+ return status;
+}
+
/* Parse input from a file and execute it */
int
@@ -1146,10 +1248,10 @@
int err = 0;
PyObject *f = PySys_GetObject(stderr);
Py_INCREF(value);
- if (f == NULL)
+ if (f == NULL) {
_PyObject_Dump(value);
- if (f == NULL)
fprintf(stderr, lost sys.stderr\n);
+ }
else {
fflush(stdout);
if (tb tb != Py_None)
@@ -1589,6 +1691,9 @@
Py_FatalError(const char *msg)
{
fprintf(stderr, Fatal Python error: %s\n, msg);
+ if (PyErr_Occurred()) {
+ PyErr_Print();
+ }
#ifdef MS_WINDOWS
OutputDebugString(Fatal Python error: );
OutputDebugString(msg);
Index: Lib/site.py
===
--- Lib/site.py (revision 58477)
+++ Lib/site.py (working copy)
@@ -402,23 +402,6 @@
(err.__class__.__name__, err))
-def installnewio():
-Install new I/O library as default.
-import io
-# Hack to avoid a nasty recursion issue when Python is invoked
-# in verbose mode: pre-import the Latin-1 and UTF-8 codecs
-from encodings import latin_1, utf_8
-# Trick so that open won't become a bound method when stored
-# as a class variable (as dumbdbm does)
-class open:
-def __new__(cls, *args, **kwds):
-return io.open(*args, **kwds)
-__builtin__.open = open
-sys.__stdin__ = sys.stdin = io.open(0, r, newline='\n')
-sys.__stdout__ = sys.stdout = io.open(1, w, newline='\n')
-sys.__stderr__ = sys.stderr = io.open(2, w, newline='\n')
-
-
def main():
abs__file__()
paths_in_sys = removeduppaths()
@@ -433,7 +416,6 @@
[issue1267] Py3K cannot run as ``python -S``
Guido van Rossum added the comment:
I've carefully checked and tested the initstdio() method. I'm sure that
I've catched every edged case. The unit tests pass w/o complains.
I've also added a PyErr_Display() call to Py_FatalError(). It's still
hard to understand an error in io.py but at least the dependency on
site.py is removed.
Very cool!
Some final suggestions:
+ Py_FatalError(Py_Initialize: can't initialize sys standard
+ streams);
Break this differently:
Py_FatalError(
Py_);
Don't call open() with keyword arg for newline=\r; open() takes
positional args too. This is done specifically to simplify life for C
code calling it. :-) Perhaps one of the PyFunction_Call(..) variants
makes it easier to call it without having to explicitly construct the
tuple for the argument list. (All this because you're leaking the
value of PyString_FromString(\n). :-)
I would change the error handling to avoid the 'finally' label, like this:
if (0) {
error:
status = -1;
Py_XDECREF(args);
}
I would add a comment see initstdio() in pythonrun.c to the
OpenWrapper class, which otherwise looks a bit mysterious (as it's not
used anywhere in the Python code).
Thanks for doing this!!!
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
Guido van Rossum wrote:
Don't call open() with keyword arg for newline=\r; open() takes
positional args too. This is done specifically to simplify life for C
code calling it. :-) Perhaps one of the PyFunction_Call(..) variants
makes it easier to call it without having to explicitly construct the
tuple for the argument list. (All this because you're leaking the
value of PyString_FromString(\n). :-)
I need a way to open a file with a specific encoding for
imp.find_module(), too. I found PyFile_FromFile(). It uses io.open but
it doesn't accept the additional arguments. I like to add another
function PyFile_FromFileEx() to the API.
PyObject *
PyFile_FromFileEx(FILE *fp, char *name, char *mode, int (*close)(FILE
*), int buffering, char *encoding, char *newline)
Some people may also miss the PyFile_FromString() function but that's a
different story. With the my new function I can create sys.stdin with:
PyFile_FromFileEx(stdin, stdin, r, fclose, -1, NULL, \n)
I kinda like it :)
I would change the error handling to avoid the 'finally' label, like this:
if (0) {
error:
status = -1;
Py_XDECREF(args);
}
Wow, that's tricky. :)
I would add a comment see initstdio() in pythonrun.c to the
OpenWrapper class, which otherwise looks a bit mysterious (as it's not
used anywhere in the Python code).
That sounds like a good idea!
The code for const char *PyTokenizer_FindEncoding(FILE *fp) and
imp.find_module() is also finished and works.
Christian
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
Christian Heimes added the comment:
The patch is a combined patch for the imp.find_module() problem and
initstdio. Both patches depend on the new PyFile_FromFileEx() function.
I hope you don't mind.
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1267
__Index: Python/pythonrun.c
===
--- Python/pythonrun.c (revision 58477)
+++ Python/pythonrun.c (working copy)
@@ -51,6 +51,7 @@
/* Forward */
static void initmain(void);
static void initsite(void);
+static int initstdio(void);
static PyObject *run_mod(mod_ty, const char *, PyObject *, PyObject *,
PyCompilerFlags *, PyArena *);
static PyObject *run_pyc_file(FILE *, const char *, PyObject *, PyObject *,
@@ -241,6 +242,9 @@
initsigs(); /* Signal handling stuff, including initintr() */
initmain(); /* Module __main__ */
+ if (initstdio() 0)
+ Py_FatalError(
+ Py_Initialize: can't initialize sys standard streams);
if (!Py_NoSiteFlag)
initsite(); /* Module site */
@@ -676,6 +680,84 @@
}
}
+/* Initialize sys.stdin, stdout, stderr and __builtin__.open */
+static int
+initstdio(void)
+{
+ PyObject *iomod=NULL, *open, *wrapper;
+ PyObject *bimod=NULL;
+ PyObject *m;
+ PyObject *std=NULL;
+ int status = 0;
+
+ /* Hack to avoid a nasty recursion issue when Python is invoked
+ in verbose mode: pre-import the Latin-1 and UTF-8 codecs */
+ if ((m = PyImport_ImportModule(encodings.utf_8)) == NULL) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(m = PyImport_ImportModule(encodings.latin_1))) {
+ goto error;
+ }
+ Py_DECREF(m);
+
+ if (!(bimod = PyImport_ImportModule(__builtin__))) {
+ goto error;
+ }
+
+ if (!(iomod = PyImport_ImportModule(io))) {
+ goto error;
+ }
+ if (!(wrapper = PyObject_GetAttrString(iomod, OpenWrapper))) {
+ goto error;
+ }
+ if (!(open = PyObject_GetAttrString(iomod, open))) {
+ goto error;
+ }
+
+ /* Set __builtin__.open */
+ if (PyObject_SetAttrString(bimod, open, wrapper) == -1) {
+ goto error;
+ }
+
+ /* Set sys.stdin */
+ if (!(std = PyFile_FromFileEx(stdin, stdin, r, fclose, -1,
+ NULL, \n))) {
+ goto error;
+ }
+ PySys_SetObject(__stdin__, std);
+ PySys_SetObject(stdin, std);
+ Py_DECREF(std);
+
+ /* Set sys.stdout */
+ if (!(std = PyFile_FromFileEx(stdout, stdout, w, fclose, -1,
+ NULL, \n))) {
+goto error;
+}
+ PySys_SetObject(__stdout__, std);
+ PySys_SetObject(stdout, std);
+ Py_DECREF(std);
+
+ /* Set sys.stderr */
+ if (!(std = PyFile_FromFileEx(stderr, stderr, w, fclose, -1,
+ NULL, \n))) {
+goto error;
+}
+PySys_SetObject(__stderr__, std);
+ PySys_SetObject(stderr, std);
+ Py_DECREF(std);
+
+if (0) {
+ error:
+status = -1;
+}
+
+ Py_XDECREF(bimod);
+ Py_XDECREF(iomod);
+ return status;
+}
+
/* Parse input from a file and execute it */
int
@@ -1146,10 +1228,10 @@
int err = 0;
PyObject *f = PySys_GetObject(stderr);
Py_INCREF(value);
- if (f == NULL)
+ if (f == NULL) {
_PyObject_Dump(value);
- if (f == NULL)
fprintf(stderr, lost sys.stderr\n);
+ }
else {
fflush(stdout);
if (tb tb != Py_None)
@@ -1589,6 +1671,9 @@
Py_FatalError(const char *msg)
{
fprintf(stderr, Fatal Python error: %s\n, msg);
+ if (PyErr_Occurred()) {
+ PyErr_Print();
+ }
#ifdef MS_WINDOWS
OutputDebugString(Fatal Python error: );
OutputDebugString(msg);
Index: Python/import.c
===
--- Python/import.c (revision 58477)
+++ Python/import.c (working copy)
@@ -91,6 +91,9 @@
/* This table is defined in config.c: */
extern struct _inittab _PyImport_Inittab[];
+/* Method from Parser/tokenizer.c */
+extern const char * PyTokenizer_FindEncoding(FILE *fp);
+
struct _inittab *PyImport_Inittab = _PyImport_Inittab;
/* these tables define the module suffixes that Python recognizes */
@@ -2558,6 +2561,7 @@
struct filedescr *fdp;
char pathname[MAXPATHLEN+1];
FILE *fp = NULL;
+ const char *encoding = NULL;
pathname[0] = '\0';
if (path == Py_None)
@@ -2566,7 +2570,11 @@
if (fdp == NULL)
return NULL;
if (fp != NULL) {
- fob = PyFile_FromFile(fp, pathname, fdp-mode, fclose);
+ encoding = PyTokenizer_FindEncoding(fp);
+ encoding = (encoding != NULL) ? encoding :
+ PyUnicode_GetDefaultEncoding();
+ fob = PyFile_FromFileEx(fp, pathname, fdp-mode, fclose,
+ -1, (char*)encoding, NULL);
if (fob == NULL) {
fclose(fp);
return NULL;
Index: Include/fileobject.h
===
--- Include/fileobject.h (revision 58477)
+++ Include/fileobject.h (working copy)
@@ -9,6 +9,9 @@
#define PY_STDIOTEXTMODE b
PyAPI_FUNC(PyObject *) PyFile_FromFile(FILE *, char *, char *, int (*)(FILE*));
+PyAPI_FUNC(PyObject *) PyFile_FromFileEx(FILE *, char *, char *,
+ int (*)(FILE *), int, char *,
+ char *);
[issue1267] Py3K cannot run as ``python -S``
Changes by Guido van Rossum: -- nosy: +gvanrossum __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1267] Py3K cannot run as ``python -S``
New submission from Brett Cannon: If Py3K is executed without importing site, it fails horribly. This is because site.py sets __builtins__.open, sys.stdout, sys.stderr, and sys.stdin. The setting of sys.stderr is especially bad as exception printing requires sys.stderr, otherwise it reports that sys.stderr was lost. This prevents debugging any exceptions thrown when site.py isn't/can't be imported (e.g., trying to debug bootstrapping importlib). -- components: Interpreter Core keywords: py3k messages: 56350 nosy: brett.cannon severity: normal status: open title: Py3K cannot run as ``python -S`` versions: Python 3.0 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1267 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
