[Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Guido van Rossum
I've got a small tweak to tokenize.py that I'd like to run by folks here. I'm working on a refactoring tool for Python 2.x-to-3.x conversion, and my approach is to build a full parse tree with annotations that show where the whitespace and comments go. I use the tokenize module to scan the input.

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Phillip J. Eby
At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote: >I've got a small tweak to tokenize.py that I'd like to run by folks here. > >I'm working on a refactoring tool for Python 2.x-to-3.x conversion, >and my approach is to build a full parse tree with annotations that >show where the whitespace and

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Guido van Rossum
Are you opposed changing tokenize? If so, why (apart from compatibility)? ISTM that it would be a good thing if it reported everything except horizontal whitespace. On 11/30/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote: > >I've got a small tw

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Fredrik Lundh
Guido van Rossum wrote: > Are you opposed changing tokenize? If so, why (apart from > compatibility)? ISTM that it would be a good thing if it reported > everything except horizontal whitespace. it would be a good thing if it could, optionally, be made to report horizontal whitespace as well.

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Phillip J. Eby
At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote: >Are you opposed changing tokenize? If so, why (apart from >compatibility)? Nothing apart from compatibility. I think you should have to explicitly request the new behavior(s), since tools (like detokenize) written to work around the old beh

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread python
> It would be trivial to add another yield to tokenize.py when > the backslah is detected +1 > I think that it should probably yield a single NL pseudo-token > whose value is a backslash followed by a newline; or perhaps it > should yield the backslash as a comment token, or as a new token. Th

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Guido van Rossum
On 11/30/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote: > >Are you opposed changing tokenize? If so, why (apart from > >compatibility)? > > Nothing apart from compatibility. I think you should have to explicitly > request the new behavior(s),

Re: [Python-Dev] Small tweak to tokenize.py?

2006-11-30 Thread Guido van Rossum
On 11/30/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Guido van Rossum wrote: > > > Are you opposed changing tokenize? If so, why (apart from > > compatibility)? ISTM that it would be a good thing if it reported > > everything except horizontal whitespace. > > it would be a good thing if it could

Re: [Python-Dev] Small tweak to tokenize.py?

2006-12-02 Thread Fredrik Lundh
Guido van Rossum wrote: >> it would be a good thing if it could, optionally, be made to report >> horizontal whitespace as well. > > It's remarkably easy to get this out of the existing API sure, but it would be even easier if I didn't have to write that code myself (last time I did that, I nee

Re: [Python-Dev] Small tweak to tokenize.py?

2006-12-02 Thread Guido van Rossum
On 12/2/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Guido van Rossum wrote: > > >> it would be a good thing if it could, optionally, be made to report > >> horizontal whitespace as well. > > > > It's remarkably easy to get this out of the existing API > > sure, but it would be even easier if I d