[issue5857] Return namedtuples from tokenize token generator

[Sergei Lebedev]

Sergei Lebedev  added the comment:

> I strongly prefer that there not be inner named tuples. 

Hi Raymond, do you still strongly prefer (row, col) to remain unnamed? If so, 
could you comment on what makes you prefer that apart from (row, col) being 
more common than (col, row)? 

Are there any performance implications/runtime costs associated with making it 
(row, col) a namedtuple?

--
nosy: +superbobry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Note that in tokenize.diff, TokenInfo should be in __all__ instead of
Token.  I agree with Raymond on the inner tuples.

--
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Committed in r72086.
Needs backporting to 2.7.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Vaibhav Mallya]

New submission from Vaibhav Mallya mally...@umich.edu:

Returning an anonymous 5-tuple seems like a suboptimal interface since
it's so easy to accidentally confuse, for example, the indices of start
and end. I've used tokenize.py for several scripts in the past few weeks
and I've always ended up writing some sort of wrapper function for
generate_tokens that names the returned tuple's fields to help me avoid
mistakes like this.

I'd like to propose the following patch that simply decorates the
generate_token function and names its return values' fields. Since it's
a namedtuple, it should be fully backwards compatible with the existing
interface, but also allow member access via 

next_token.type
next_token.string
next_token.start.row, next_token.start.col
next_token.end.row, next_token.end.col
next_token.line

If this seems like a reasonable way to do things, I'd be happy to submit
relevant doc patches as well as the corresponding patch for 3.0.

--
components: Library (Lib)
files: mallyvai_tokenize.patch
keywords: patch
messages: 86691
nosy: mallyvai
severity: normal
status: open
title: Return namedtuples from tokenize token generator
type: feature request
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1
Added file: http://bugs.python.org/file13797/mallyvai_tokenize.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Comments on the earlier patch:
* No need for an inner row/col namedtuple.  That would add little value.
* The name Token is already used in the module for a different purpose.
* The wrapper is nice looking, but it is better to in-line this patch.

Attaching an alternate patch.

--
keywords: +needs review -patch
nosy: +rhettinger
versions:  -Python 2.6, Python 3.0
Added file: http://bugs.python.org/file13799/tokenize.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Vaibhav Mallya]

Vaibhav Mallya mally...@umich.edu added the comment:

Well, the reason I put in the inner row/col namedtuple initially was
because the first mistake I made with the original module was mixing up
the row/col indices for a particular case. It certainly caused all sorts
of weird headaches. :o)

I mean, it seems like there's no real reason it should be (row,col)
instead of (col,row) in the returned tuple; that is, it feels like the
ordering is arbitrary in and of itself.

I really feel that allowing for start.row and start.col would make the
interface completely explicit and valid semantically.

Agreed with the other two points, however.

Also, I take it there's going to be a need for an addendum to the test
suite, since the interface is being modified?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

I strongly prefer that there not be inner named tuples.  That is going
overboard.  FWIW, row/col order is very common convention especially
when the row refers to a line number in a text block and column refers
to a character position within the row.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5857] Return namedtuples from tokenize token generator

[Raymond Hettinger]

Changes by Raymond Hettinger rhettin...@users.sourceforge.net:


--
assignee:  - rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5857
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com