Serhiy Storchaka <storchaka+cpyt...@gmail.com> added the comment:
NEWLINE is not a newline. It is the NEWLINE token. And it is generated at the
end of file.
$ echo 'print("a");print("b")' | ./python -m tokenize
1,0-1,5: NAME 'print'
1,5-1,6: OP '('
1,6-1,9: STRING '"a"'
1,9-1,10: OP ')'
1,10-1,11: OP ';'
1,11-1,16: NAME 'print'
1,16-1,17: OP '('
1,17-1,20: STRING '"b"'
1,20-1,21: OP ')'
1,21-1,22: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
----------
nosy: +serhiy.storchaka
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33766>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
