[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[Florent Xicluna]

New submission from Florent Xicluna :

I've hit this issue while playing with tokenize for the pep8.py module.

The tokenize detect_encoding() should report SyntaxError when the encoding is 
improperly declared.

However it raises a LookupError in some cases.

$ ./python -m tokenize Lib/test/bad_coding2.py 
unexpected error: unknown encoding: utf8-sig
Traceback (most recent call last):
  File "./Lib/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "./Lib/runpy.py", line 75, in _run_code
exec(code, run_globals)
  File "./Lib/tokenize.py", line 686, in 
main()
  File "./Lib/tokenize.py", line 656, in main
tokens = list(tokenize(f.readline))
  File "./Lib/tokenize.py", line 489, in _tokenize
line = line.decode(encoding)
LookupError: unknown encoding: utf8-sig

--
components: Library (Lib)
messages: 162205
nosy: flox
priority: normal
severity: normal
stage: needs patch
status: open
title: detect_encoding should fail with SyntaxError on invalid encoding
type: behavior
versions: Python 3.1, Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[Florent Xicluna]

Florent Xicluna  added the comment:

This patch seems to fix the issue.

--
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file25805/issue14990_detect_encoding.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[STINNER Victor]

STINNER Victor  added the comment:

The patch is correct according to the PEP 263:

If a source file uses both the UTF-8 BOM mark signature and a
magic encoding comment, the only allowed encoding for the comment
is 'utf-8'.  Any other encoding will cause an error.

The fix should also be applied to 3.2.

(Note: Python 3.1 doesn't accept bugfixes anymore.)

--
components: +Unicode
nosy: +ezio.melotti, haypo
versions:  -Python 3.1
Added file: http://bugs.python.org/file25825/detect_encoding.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[Florent Xicluna]

Florent Xicluna  added the comment:

It should raise a SyntaxError, if coding is 'utf8'.
I don't agree with the last patch proposed.

If the import report a SyntaxError, 'tokenize' should do the same.

$ ./python Lib/test/bad_coding2.py
  File "Lib/test/bad_coding2.py", line 1
SyntaxError: encoding problem: utf-8

and it complies strictly with PEP263.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file25825/detect_encoding.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[STINNER Victor]

STINNER Victor  added the comment:

Oops, I didn't want to attach my patch to the issue. Mine is wrong, whereas 
yours is the right fix :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 5020afc0b7c9 by Florent Xicluna in branch '3.2':
Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding 
declaration.
http://hg.python.org/cpython/rev/5020afc0b7c9

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14990] detect_encoding should fail with SyntaxError on invalid encoding

[Florent Xicluna]

Florent Xicluna  added the comment:

Thanks. Fixed in trunk too, changeset b4322ad1fec4

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com