[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Jean-Michel Fauth added the comment: Quick feedback from a Windows user. I made a few more tests with the freshly installed Pyton 3.1a1. The compile() function is running very well. As a side effect, it now possible to write an "execfile()" without any problem. Tests with execfile() versions r

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: > Yes, I could, but I think this is not an IDLE issue > (...) > File "uuu.py", line 8, in > ii.runsource(source) > (...) > File "c:\python30\lib\codeop.py", line 70, in _maybe_compile > for line in source.split("\n"): > TypeError: Type str doesn't

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Jean-Michel Fauth added the comment: > Victor Yes, I could, but I think this is not an IDLE issue, I'm just using IDLE to illustrate the problem. I got the same when I'm working from within an editor or with my interactive interpreter I wrote for the fun. Code in the editor: # -*- coding: cp1

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: @jmfauth: Can you open a different issue for the IDLE issue? -- ___ Python tracker ___ ___ Python-bu

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Jean-Michel Fauth added the comment: When I was preparing some test examples to be submitted here. I noticed the module codeop.py used by the InteractiveInterpreter, does not like byte strings very much. IDLE, Python 3.0.1, winxp sp2 >>> source = b'print(999)' >>> compile(source, '', 'exec')

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Brett Cannon added the comment: On Tue, Mar 24, 2009 at 09:24, Jean-Michel Fauth wrote: > > Jean-Michel Fauth added the comment: > > I'm glad to have discovered this topic. I bumped into something similar > when I toyed with an interactive interpreter. > > from code import InteractiveInterpret

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Jean-Michel Fauth added the comment: I'm glad to have discovered this topic. I bumped into something similar when I toyed with an interactive interpreter. from code import InteractiveInterpreter ii = InteractiveInterpreter() source = ... ii.runsource(source) What should be the encoding and/or

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: > Should this be backported? It's the r70113 (not the 70112). I see that pitrou backported the fix to 3.0.x. I think that it's enough, 2.x doesn't require the fix. ___ Python tracker

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: Should this be backported? ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: Fixed in r70112. -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Pytho

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: I'll deal with it eventually. -- assignee: -> benjamin.peterson ___ Python tracker ___ ___ Pytho

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: > I think that feature is ok to not support. Yeah! Anyone to review and/or commit the last patch? -- versions: +Python 3.1 ___ Python tracker

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: On Tue, Feb 17, 2009 at 6:22 PM, STINNER Victor wrote: > I don't understand the change in source_as_string(). Except of that, > it looks correct. Py_CFFLAGS_SOURCE_IS_UTF8 is already set in compile(). > >> The one problem is that it causes test_coding to f

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: @benjamin.peterson: I don't see you changes... I read both patches carrefuly and I saw: - source_as_string(): remove "cf.cf_flags = supplied_flags | PyCF_SOURCE_IS_UTF8;" - rename PyTokenizer_FromUnicode() to PyTokenizer_FromUTF8() - update NEWS I don't un

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: Here's another patch. The one problem is that it causes test_coding to fail because coding cookies are ignored and not checked to see if they are the same as the encoding of the file. It seems fine to me to just skip this test. -- keywords: +needs re

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: On Thu, Jan 29, 2009 at 5:13 PM, STINNER Victor wrote: > Ok, I created a new function PyTokenizer_FromUnicode(). I > choosed "FromUnicode" because the string is encoded in unicode (as > UTF-8, even if it's not the wchar_t* type). How about PyTokenizer_FromU

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Changes by STINNER Victor : Removed file: http://bugs.python.org/file12771/tokenizer_ignore_cookie-3.patch ___ Python tracker ___ ___ Python-bu

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: > I don't like the change of API to PyTokenizer_FromString. > I would prefer another function like PyTokenizer_IgnoreCodingCookie() Ok, I created a new function PyTokenizer_FromUnicode(). I choosed "FromUnicode" because the string is encoded in unicode (as

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Amaury Forgeot d'Arc added the comment: Note that PyTokenizer_FromString is not an API function: it's not marked with PyAPI_FUNC, and not exported from the windows DLL. ___ Python tracker ___

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Benjamin Peterson added the comment: I don't like the change of API to PyTokenizer_FromString. I would prefer another function like PyTokenizer_IgnoreCodingCookie() blows up when parsing has already started. The (char *) cast in PyTokenizer_FromString is unneeded. You need to indent the "else"

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: Ping! Can anyone review my patch? ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Brett Cannon added the comment: I will see when I can get to it (other stuff is taking priority). Not going to assign to myself quite yet in case someone wants to beat me to this. I won't lose track since I created the bug and have a saved query to look at all bugs I make. _

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Changes by STINNER Victor : Removed file: http://bugs.python.org/file12589/tokenizer_ignore_cookie-2.patch ___ Python tracker ___ ___ Python-bu

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: New version of my patch: add a regression test. @brett.cannon: Could you review my patch? Added file: http://bugs.python.org/file12771/tokenizer_ignore_cookie-3.patch ___ Python tracker _

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: > With my patch, the first Brett's example displays: >... >” = '”' > ^ >SyntaxError: invalid character in identifier > > The error cursor is not at the right column (bug related to the issue > 2382 or introduced by my patch?). I tried p

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Changes by STINNER Victor : Removed file: http://bugs.python.org/file12588/tokenizer_ignore_cookie.patch ___ Python tracker ___ ___ Python-bugs

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: Oops, I attached the wrong file :-p Added file: http://bugs.python.org/file12589/tokenizer_ignore_cookie-2.patch ___ Python tracker ___

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

STINNER Victor added the comment: The function decode_str() (Parser/tokenizer.c) is responsible to detect the encoding using the BOM or the cookie ("coding: xxx"). decode_str() reencodes also the text to utf-8 if the encoding is different than utf-8. I think that we can just skip this functio

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Brett Cannon added the comment: So explicitly converting to UTF-8 didn't work, or at least as simply as I had hoped. ___ Python tracker ___ ___

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Brett Cannon added the comment: Here is what I have found out so far. Python/bltinmodule.c:builtin_compile takes in a PyObject and gets the char * representation of that object and passes it to Python/pythonrun.c:Py_CompileStringFlags. Unfortunately no other information is passed along in the ca

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Changes by John Machin : -- nosy: +sjmachin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

Amaury Forgeot d'Arc added the comment: Issue4742 is similar issue: >>> source = b"# coding=cp1252\n\x94 = '\x94'".decode('cp1252') >>> compile(source, '', 'exec') Traceback (most recent call last): File "", line 1, in File "", line 0 SyntaxError: unknown encoding: cp1252 The real error h

[issue4626] compile() doesn't ignore the source encoding when a string is passed in

New submission from Brett Cannon <[EMAIL PROTECTED]>: When compile() is called with a string it is a reasonable assumption that it has already been decoded. But this is not in fact the case and leads to errors when trying to use non-ASCII identifiers:: >>> source = "# coding=latin-1\n\u00c6 = '