I've got a small tweak to tokenize.py that I'd like to run by folks here.
I'm working on a refactoring tool for Python 2.x-to-3.x conversion,
and my approach is to build a full parse tree with annotations that
show where the whitespace and comments go. I use the tokenize module
to scan the input.
At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote:
>I've got a small tweak to tokenize.py that I'd like to run by folks here.
>
>I'm working on a refactoring tool for Python 2.x-to-3.x conversion,
>and my approach is to build a full parse tree with annotations that
>show where the whitespace and
Are you opposed changing tokenize? If so, why (apart from
compatibility)? ISTM that it would be a good thing if it reported
everything except horizontal whitespace.
On 11/30/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote:
> >I've got a small tw
Guido van Rossum wrote:
> Are you opposed changing tokenize? If so, why (apart from
> compatibility)? ISTM that it would be a good thing if it reported
> everything except horizontal whitespace.
it would be a good thing if it could, optionally, be made to report
horizontal whitespace as well.
At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
>Are you opposed changing tokenize? If so, why (apart from
>compatibility)?
Nothing apart from compatibility. I think you should have to explicitly
request the new behavior(s), since tools (like detokenize) written to work
around the old beh
> It would be trivial to add another yield to tokenize.py when
> the backslah is detected
+1
> I think that it should probably yield a single NL pseudo-token
> whose value is a backslash followed by a newline; or perhaps it
> should yield the backslash as a comment token, or as a new token.
Th
On 11/30/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote:
> At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
> >Are you opposed changing tokenize? If so, why (apart from
> >compatibility)?
>
> Nothing apart from compatibility. I think you should have to explicitly
> request the new behavior(s),
On 11/30/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
>
> > Are you opposed changing tokenize? If so, why (apart from
> > compatibility)? ISTM that it would be a good thing if it reported
> > everything except horizontal whitespace.
>
> it would be a good thing if it could
Guido van Rossum wrote:
>> it would be a good thing if it could, optionally, be made to report
>> horizontal whitespace as well.
>
> It's remarkably easy to get this out of the existing API
sure, but it would be even easier if I didn't have to write that code
myself (last time I did that, I nee
On 12/2/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
>
> >> it would be a good thing if it could, optionally, be made to report
> >> horizontal whitespace as well.
> >
> > It's remarkably easy to get this out of the existing API
>
> sure, but it would be even easier if I d
10 matches
Mail list logo