[issue25311] Add f-string support to tokenize.py

Roundup Robot added the comment: New changeset 21f6c4378846 by Eric V. Smith in branch 'default': Issue 25311: Add support for f-strings to tokenize.py. Also added some comments to explain what's happening, since it's not so obvious. https://hg.python.org/cpython/rev/21f6c4378846 --

[issue25311] Add f-string support to tokenize.py

Changes by Eric V. Smith : -- keywords: -patch stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: I've fixed this particular problem, but the tokenize module definitely has some other issues. It recompiles regexes very often when it doesn't need to, it treats single- and triple-quoted strings differently (leading to some code bloat), etc. I may open

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: This patch cleans up string matching in tokenize.py, and adds f-string support. -- Added file: http://bugs.python.org/file40821/issue25311-1.diff ___ Python tracker

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: Multi-line string tests were added in changeset 91c44dc35dfd. That will make changes for this issue safer. Updated patch to come. -- ___ Python tracker

[issue25311] Add f-string support to tokenize.py

Changes by Ankit Baruah : -- nosy: +@nkit ___ Python tracker ___ ___

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: Oops, make that 80 combinations (I forgot the various 'fb' ones): ['B', 'BF', 'BFR', 'BFr', 'BR', 'BRF', 'BRf', 'Bf', 'BfR', 'Bfr', 'Br', 'BrF', 'Brf', 'F', 'FB', 'FBR', 'FBr', 'FR', 'FRB', 'FRb', 'Fb', 'FbR', 'Fbr', 'Fr', 'FrB', 'Frb', 'R', 'RB', 'RBF',

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: I think the best way to approach this is to generate (in code) all of the places where string prefixes appear. There's StringPrefix, endpats, triple_quotes, and single_quoted. With the currently valid combinations of f, b, r, and u, I count 24 combinations:

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: My first attempt. Many more tests are needed. I'm going to need to spend some time trying to figure out how parts of tokenize.py actually works. I'm not sure, for example, that endpats is initialized correctly. There definitely aren't enough tests, since if I

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: Yes, both 'fr' and 'rf' need to be supported (and all upper/lower variants). And in the future, maybe 'fb' (and 'rfb', 'bfr', ...). Unfortunately, the regex doesn't scale well for all of the combinations. -- ___

[issue25311] Add f-string support to tokenize.py

Martin Panter added the comment: Thanks for the patch. Do you want to try adding a test case. See TokenizeTest.test_string() at /Lib/test/test_tokenize.py:189 for a guide, though I would suggest a new test_fstring() method. Also, F-strings can be combined with the raw string syntax. I wonder

[issue25311] Add f-string support to tokenize.py

Nan Wu added the comment: Added 'f'/'F' to the StringPrefix regex and also update the quote dictionary. -- keywords: +patch nosy: +Nan Wu Added file: http://bugs.python.org/file40712/tokenize.patch ___ Python tracker

[issue25311] Add f-string support to tokenize.py

New submission from Stefan Krah: I think tokenize.py needs to be updated to support f-strings. BTW, the f-string implementation seems to be incredibly robust. Nice work! -- components: Library (Lib) messages: 252274 nosy: eric.smith, skrah priority: normal severity: normal stage:

[issue25311] Add f-string support to tokenize.py

Eric V. Smith added the comment: Thanks for noticing tokenize.py. And thanks for the kind note! -- assignee: -> eric.smith ___ Python tracker ___

[issue25311] Add f-string support to tokenize.py

Martin Panter added the comment: I was just about to make the same bug report :) I guess it would be fine to tokenize F-strings as the same string objects as others, it probably just needs adding an F to the right regular expression. $ ./python -btWall -m tokenize "string" 1,0-1,8: