[issue8478] tokenize.untokenize first token missing failure case

[Berker Peksag]

Change by Berker Peksag :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Roundup Robot]

Roundup Robot added the comment:

New changeset 8d6dd02a973f by Terry Jan Reedy in branch '3.3':
Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The
http://hg.python.org/cpython/rev/8d6dd02a973f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Roundup Robot]

Roundup Robot added the comment:

New changeset c2517a37c13a by Terry Jan Reedy in branch '2.7':
Issue #8478: Untokenizer.compat now processes first token from iterator input.
http://hg.python.org/cpython/rev/c2517a37c13a

New changeset b6d6ca792b64 by Terry Jan Reedy in branch '3.3':
Issue #8478: Untokenizer.compat now processes first token from iterator input.
http://hg.python.org/cpython/rev/b6d6ca792b64

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Roundup Robot]

Roundup Robot added the comment:

New changeset c896d292080a by Terry Jan Reedy in branch '2.7':
Untokenize: An logically incorrect assert tested user input validity.
http://hg.python.org/cpython/rev/c896d292080a

New changeset 51e5a89afb3b by Terry Jan Reedy in branch '3.3':
Untokenize: An logically incorrect assert tested user input validity.
http://hg.python.org/cpython/rev/51e5a89afb3b

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Terry J. Reedy]

Changes by Terry J. Reedy :


--
assignee:  -> terry.reedy
nosy: +terry.reedy
stage:  -> patch review
versions: +Python 2.7, Python 3.3, Python 3.4 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Georg Brandl]

Changes by Georg Brandl :


--
assignee: georg.brandl -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Thomas Kluyver]

Thomas Kluyver added the comment:

#16224 appears to be a duplicate.

There seem to be several quite major issues with untokenize - see also #12691 - 
with patches made to fix them. Is there anything I can do to help push these 
forwards?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Thomas Kluyver]

Changes by Thomas Kluyver :


--
nosy: +takluyver

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Eric Snow]

Changes by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +Arfrever

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Georg Brandl]

Georg Brandl added the comment:

Attaching patch.  Actually both versions of untokenize() were broken; the 
version used for "full input" (5-tuples) had a flipped inequality sign in an 
assert.

Other changes in the patch:

* Docs fixed to describe both modes
* Tests fixed to exercise both modes

--
keywords: +patch
Added file: http://bugs.python.org/file27452/untokenize.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[Georg Brandl]

Changes by Georg Brandl :


--
assignee:  -> georg.brandl
nosy: +georg.brandl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[rb]

Changes by rb :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8478] tokenize.untokenize first token missing failure case

[rb]

New submission from rb :

When altering tokens and thus not providing token location information, 
tokenize.untokenize sometimes misses out the first token. Failure case below.

Expected output: 'import foo ,bar\n'
Actual output: 'foo ,bar\n'

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:43:55) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import StringIO, tokenize
>>> 
>>> def strip(iterable):
... for t_type, t_str, (srow, scol), (erow, ecol), line in iterable:
... yield t_type, t_str
... 
>>> source = StringIO.StringIO('import foo, bar\n')
>>> print 
>>> repr(tokenize.untokenize(strip(tokenize.generate_tokens(source.readline
'foo ,bar \n'
>>> source.seek(0)
>>> print repr(tokenize.untokenize(tokenize.generate_tokens(source.readline)))
'import foo, bar\n'
>>>

--
components: Library (Lib)
messages: 103799
nosy: rb
severity: normal
status: open
title: tokenize.untokenize first token missing failure case
versions: Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com