Gareth Rees added the comment: This morning I noticed that I had forgotten to update the library reference, and I also noticed two more problems to add to the list above:
6. Although Lib/test/test_tokenize.py looks like it contains tests for backslash-newline handling, these tests are ineffective. Here they are: >>> roundtrip("x=1+\\\\n" ... "1\\n" ... "# This is a comment\\\\n" ... "# This also\\n") True >>> roundtrip("# Comment \\\\nx = 0") True There are two problems here: (i) because of the double string escaping, these are not backslash-newline, they are backslash-n. (ii) the roundtrip() test is too weak to detect this problem: tokenize() outputs an ERRORTOKEN for the backslash and untokenize() restores it. So the round-trip property is satisfied. 7. Problem 6 shows the difficulty of using doctests for this kind of test. It would be easier to ensure the correctness of these tests if the docstring was read from a separate file, so that at least the tests only need one level of string escaping. I fixed problem 6 by updating these tests to use dump_tokens() instead of roundtrip(). I have not fixed problem 7 (like 4 and 5, I can leave it for another issue). Revised patch attached. ---------- Added file: http://bugs.python.org/file33924/Issue12691.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue12691> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com