STINNER Victor <victor.stin...@haypocalc.com> added the comment: The function decode_str() (Parser/tokenizer.c) is responsible to detect the encoding using the BOM or the cookie ("coding: xxx"). decode_str() reencodes also the text to utf-8 if the encoding is different than utf-8. I think that we can just skip this function if the input text is already unicode (utf-8). Attached patch implements this idea.
The patch introduces a new compiler flag (PyCF_IGNORE_COOKIE) and a new parser flag (PyPARSE_IGNORE_COOKIE). The new compiler flag is set by source_as_string() when the input is a PyUnicode object. "Ignore cookie" is maybe not the best name for this flag. With my patch, the first Brett's example displays: $ ./python com2.py Traceback (most recent call last): File "com2.py", line 3, in <module> compile(source, '<test>', 'exec') File "<test>", line 2 ” = '”' ^ SyntaxError: invalid character in identifier The error cursor is not at the right column (bug related to the issue 2382 or introduced by my patch?). The patch changes the public API: PyTokenizer_FromString() prototype changed to get a new argument. I don't like changing public API. The new argument should be a bit vector (flags) instead of a single bit (ignore_cookie). We can avoid changing the public API by creating a new function (eg. "PyTokenizer_FromUnicode" ;-)). There are some old PyPARSE_xxx constants in Include/parsetok.h that might be removed. PyPARSE_WITH_IS_KEYWORD value is 3 which is strange since flags is a bit vector (changed with | and tested by &). But PyPARSE_WITH_IS_KEYWORD is a dead constant (written in #if 0...#endif). ---------- keywords: +patch nosy: +haypo Added file: http://bugs.python.org/file12588/tokenizer_ignore_cookie.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue4626> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com