[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: yep good call terry, not getting any response from the autopep8 developer, and i believe it was down to a loop where the text is being thrown line-by-line at tokenize and it was losing critical state information. so... not a bug in tokenize.

[issue34428] tokenize

Terry J. Reedy added the comment: Luke: If and when you have legal code that fails when run with purely with CPython, you can re-open or try an new issue. -- nosy: +serhiy.storchaka, terry.reedy resolution: -> third party stage: -> resolved status: open -> closed versions: +Python

[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: ahh darn-it, autopep8 is passing in tokens line-by-line, to be parsed one at a time oh and of course it's losing state information that tokenizer critically relies on. i *think* that's what's going on so it's highly unlikely to be a

[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: wtf??? neither can i import io import tokenize text = r'''\ ( r"\(") ( "\(") ''' string_io = io.StringIO(text) tokens = list( tokenize.generate_tokens(string_io.readline) ) print (tokens) works perfectly. ok a i bet you it's

[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: regular expressions are not something i am familiar or comfortable with (never have been: the patterns are too dense). however REMOVING "Bracket" from the regular expression(s) for PseudoToken "fixes" the problem. some debug print statements

[issue34428] tokenize

Serhiy Storchaka added the comment: I can't reproduce. >>> import tokenize >>> list(tokenize.generate_tokens(iter(['(\n', r'"\(")']).__next__)) [TokenInfo(type=53 (OP), string='(', start=(1, 0), end=(1, 1), line='(\n'), TokenInfo(type=56 (NL), string='\n', start=(1, 1), end=(1, 2),

[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: python2.7 and 3.5 also has exact same issue. -- versions: +Python 2.7, Python 3.5 ___ Python tracker ___

[issue34428] tokenize

Luke Kenneth Casson Leighton added the comment: these two line also pass (do not throw an exception): co = re.compile( r"\(") the code that fails may be further reduced to the following: ( "\(") -- ___ Python tracker

[issue34428] tokenize

New submission from Luke Kenneth Casson Leighton : https://github.com/hhatto/autopep8/issues/414 the following two lines of code are not parseable by tokenize.py: co = re.compile( "\(") the combination of: * being split on two lines * having a backslash inside quotes * having a