On Apr 4, 4:38 pm, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> George Sakkis wrote:
> >> If it was a bug it has to violate a functional requirement. I can't
> >> see which one.
>
> > Perhaps it's not a functional requirement but it came up as a real
> > problem on a source colorizer I use. I count o
George Sakkis wrote:
>> If it was a bug it has to violate a functional requirement. I can't
>> see which one.
>
> Perhaps it's not a functional requirement but it came up as a real
> problem on a source colorizer I use. I count on newlines generating
> token.NEWLINE or tokenize.NL tokens in order
On Apr 4, 3:18 pm, Kay Schluehr <[EMAIL PROTECTED]> wrote:
> I guess it's just an artifact of handling line continuations within
> expressions where a different rule is applied. For compilation
> purposes both the newlines within expressions as well as the comments
> are irrelevant. There are even
On 4 Apr., 18:22, George Sakkis <[EMAIL PROTECTED]> wrote:
> The tokenize.generate_tokens function seems to handle in a context-
> sensitive manner the new line after a comment:
>
> >>> from StringIO import StringIO
> >>> from tokenize import generate_tokens
>
> >>> text = '''
>
> ... # hello world
The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:
>>> from StringIO import StringIO
>>> from tokenize import generate_tokens
>>>
>>> text = '''
... # hello world
... x = (
... # hello world
... )
... '''
>>>
>>> for t in generate_tok
