[issue24447] tab indentation breaks in tokenize.untokenize

[Terry J. Reedy]

Changes by Terry J. Reedy :


--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> tokenize/untokenize roundtrip fails with tabs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24447] tab indentation breaks in tokenize.untokenize

[Dingyuan Wang]

New submission from Dingyuan Wang:

If a script uses tabs for indentation, tokenize.untokenize won't restore 
original indentation correctly from the second line of the indentation level, 
and thus breaks the file.

This affects all Python versions.

Test code:

python2 -c 'import sys, tokenize; 
sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))'
 < tab.py

python3 -c 'import sys, tokenize; 
sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))'
 < tab.py

Out:

def foo():
pass
 pass
 if 1:
pass
  pass

--
components: Library (Lib)
files: tab.py
messages: 245333
nosy: gumblex
priority: normal
severity: normal
status: open
title: tab indentation breaks in tokenize.untokenize
type: behavior
versions: Python 3.4
Added file: http://bugs.python.org/file39704/tab.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com