[issue12943] tokenize: add python -m tokenize support back

2011-10-07 Thread Meador Inge 

Changes by Meador Inge :


--
assignee: docs@python -> meador.inge
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-10-07 Thread Roundup Robot 

Roundup Robot  added the comment:

New changeset 85254bb6c1c6 by Meador Inge in branch 'default':
Issue #12943: python -m tokenize support has been added to tokenize.
http://hg.python.org/cpython/rev/85254bb6c1c6

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-10-07 Thread Meador Inge 

Meador Inge  added the comment:

Éric, thanks.  I fixed most of your points.  And thanks to everyone that 
reviewed this.  Much appreciated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-10-06 Thread Éric Araujo 

Éric Araujo  added the comment:

I made a few last remarks on Rietveld; feel free to address or ignore them and 
commit right away.

--
nosy: +eric.araujo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-10-02 Thread Meador Inge 

Meador Inge  added the comment:

Fixed a few more nits pointed out in review.

--
Added file: http://bugs.python.org/file23304/issue12943-6.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-22 Thread Georg Brandl 

Georg Brandl  added the comment:

I would say this looks good now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-22 Thread Meador Inge 

Meador Inge  added the comment:

> I don't think the help option needs to be documented, it will document 
> itself.

Normally I would document it anyway, but in this case there is only the
one option.  So, I dropped it.

> An additional suggestion is to catch errors on tokenizing and exiting 
> with a non-zero exit status in these cases.

I like that idea.  Sample output:

$ python -m tokenize foo.py
foo.py: error: unknown encoding: meador
$ python -m tokenize foo.py
foo.py:7:2: error: unindent does not match any outer indentation level
$ ./python -m tokenize bogus.py
error: [Errno 2] No such file or directory: 'bogus.py'
$ python -m tokenize foo.py
$ echo '[1,' | ./python -m tokenize
1,0-1,1:OP '['
1,1-1,2:NUMBER '1'
1,2-1,3:OP ','
1,3-1,4:NL '\n'   
:2:0: error: EOF in multi-line statement

Attached is the latest patch implementing these suggestions.

--
Added file: http://bugs.python.org/file23228/issue12943-5.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-22 Thread Georg Brandl 

Georg Brandl  added the comment:

I don't think the help option needs to be documented, it will document itself.

An additional suggestion is to catch errors on tokenizing and exiting with a 
non-zero exit status in these cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-19 Thread Meador Inge 

Meador Inge  added the comment:

Updated patch which adds and documents a '-h,--help' option.  This option
was suggested during code review.

--
Added file: http://bugs.python.org/file23196/issue12943-4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-19 Thread Meador Inge 

Meador Inge  added the comment:

Updated patch fixing some issues pointed out by Ezio on Rietveld.

--
Added file: http://bugs.python.org/file23193/issue12943-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-17 Thread Meador Inge 

Meador Inge  added the comment:

> Did you get commit rights already?

I have not.  I still need to submit a contributor agreement as well.  I
plan to fax that today.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-17 Thread Georg Brandl 

Georg Brandl  added the comment:

Looks good to me. Did you get commit rights already?

--
nosy: +georg.brandl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-17 Thread Meador Inge 

Meador Inge  added the comment:

v2 patch which addresses comments made by merwok via rietveld.

--
Added file: http://bugs.python.org/file23184/issue12943-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-10 Thread Meador Inge 

Meador Inge  added the comment:

Patch against tip.

--
assignee:  -> docs@python
components: +Documentation
keywords: +needs review, patch
nosy: +docs@python
stage: needs patch -> patch review
Added file: http://bugs.python.org/file23127/issue12943.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12943] tokenize: add python -m tokenize support back

2011-09-08 Thread Meador Inge 

New submission from Meador Inge :

In 2.x, 'python -m tokenize' worked great:

[meadori@motherbrain cpython]$ python2.7 -m tokenize test.py
1,0-1,5:NAME'print'
1,6-1,21:   STRING  '"Hello, World!"'
1,21-1,22:  NEWLINE '\n'
2,0-2,0:ENDMARKER   ''

In 3.x, however, the functionality has been removed and replaced with
some hard-wired test code:

[meadori@motherbrain cpython]$ python3 -m tokenize test.py
TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3),
line='def parseline(self, line):')
TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1,
13), line='def parseline(self, line):')
TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14),
line='def parseline(self, line):')
...

The functionality was removed here [1], but with no explanation.  Let's add it 
back and document the functionality this time around.

[1] http://hg.python.org/cpython/rev/51e24512e305/

--
components: Library (Lib)
messages: 143752
nosy: meadori
priority: normal
severity: normal
stage: needs patch
status: open
title: tokenize: add python -m tokenize support back
type: feature request
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com