Martin v. Löwis <[EMAIL PROTECTED]> added the comment: In 3k, the tokenize module should definitely return strings, and, in doing so, it should definitely consider the encoding declaration (and also the default encoding in absence of the encoding declaration).
For 2.6, I wouldn't mind if it were changed incompatibly so that it returns Unicode strings, or else that it parses in Unicode, and then encodes back to the source encoding before returning anything. ____________________________________ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue719888> ____________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
