[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Documentation task added as issue #25179. Thanks to Martin for the great code reviews. -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

Roundup Robot added the comment: New changeset a10d37f04569 by Eric V. Smith in branch 'default': Issue #24965: Implement PEP 498 "Literal String Interpolation". Documentation is still needed, I'll open an issue for that. https://hg.python.org/cpython/rev/a10d37f04569 --

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: I changed the generated code to call: format(x [, spec]) instead of: x.__format__(spec) The reason is that the correct way to call __format__ is actually: type(x).__format__(x, spec) That is, the __format__ lookup is done on the type, not the instance. From

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Cleaned up the error handling in fstring_expression_compile so it's easier to verify and more robust in the face of future changes. Added a test for an un-doubled '}', which is an error in a top-level literal (and ends a nested expression). Modified existing

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Hopefully the last version. -- Added file: http://bugs.python.org/file40495/pep-498-8.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: > Martin Panter added the comment: > > Another strange error message (though maybe the new test changes you > mentioned caught this): > f'{3:{10}' # Actually missing a closing bracket '}' > File "", line 1 > SyntaxError: f-string: unexpected '}' Yes,

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: Another strange error message (though maybe the new test changes you mentioned caught this): >>> f'{3:{10}' # Actually missing a closing bracket '}' File "", line 1 SyntaxError: f-string: unexpected '}' -- ___

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: I left a few more comments on Reitveld. Checking the error messages does make me feel a lot more comfortable though. -- ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: I rewrote the format_spec parser to recursively call the f-string parser, so any oddness in what's allowed in a format_spec is gone. It took way longer than I thought, but the code is better for it. -- Added file:

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Simplified error handling, fixed 2 memory leaks. All tests now pass with no leaks. This should be the final version. -- Added file: http://bugs.python.org/file40484/pep-498-7.diff ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Yury Selivanov : -- nosy: +elvis ___ Python tracker ___ ___ Python-bugs-list

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: Regarding the null terminator, I was mainly smoke testing your code. :) Maybe it would be too hard to support properly. Although I could imagine someone doing things like this: >>> d = {b"key\x00": "value"} >>> f"key={d[b'key\x00']}" # Oops, escape code at

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: I'll probably ensure that all of the parsing errors contain "format string" or "f-string" or similar. That way the regex check is easier, and the user can search for it more easily. It remains to be seen how these are referenced in the documentation.

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: After discussing it with Guido, I've removed the ability to combine 'f' with 'u'. -- ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Jelle Zijlstra added the comment: Thanks! Here are a few more cases I came across with the existing implementation: >>> f"{'a\\'b'}" File "", line 1 SyntaxError: missing '}' in format string expression I believe this is valid and should produce "a'b". >>> f"{x!s!s}" File "", line 1

[issue24965] Implement PEP 498: Literal String Formatting

Jelle Zijlstra added the comment: Is this behavior intentional? >>> str = len >>> x = 'foo' >>> f'{x!s}' '3' >>> '{!s}'.format(x) 'foo' Or similarly: >>> import builtins >>> del builtins.repr >>> f'{x!r}' Traceback (most recent call last): File "", line 1, in NameError: name 'repr' is not

[issue24965] Implement PEP 498: Literal String Formatting

Jelle Zijlstra added the comment: I've started working on implementing this feature in Cython and I'd like to confirm a few edge cases: - f'{ {1: 2\N{RIGHT CURLY BRACKET}[1]}' == '2' (string escape rules work even within the expressions) - f'{ '''foo''' }' is a syntax error - f'{ """foo

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: This patch fixes triple-quoted strings, plus a few bugs. I'm going to commit it tomorrow, barring any unforeseen issues. -- Added file: http://bugs.python.org/file40447/pep-498-5.diff ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Yes, Jelle, you are correct in all 3 cases. Remember that the steps are to extract the string from the source code, decode backslash escapes, and only then treat it as an f-string. For the first case, without the 'f' prefix: '{ {1: 2\N{RIGHT CURLY

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Agreed on checking the error messages better. Especially since even the simplest of errors (like leaving out a quote) results in a syntax error in parsing the string, not parsing inside the f-string. I'll look at it eventually. --

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: Regarding wrong error messages, I’ve learnt the hard way that it is often best to use assertRaisesRegex() instead of assertRaises(), to ensure that the actual exception you have in mind is being triggered, rather than a typo or something. Though that might

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: This one has been fixed: >>> f"{'a\\'b'}" "a'b" This one was a bug that I previously fixed, that Martin pointed out: >>> f"{x!s!s}" File "", line 1 SyntaxError: invalid character following conversion character And this is the same bug: >>> f"{x!s{y}}" File

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Both of those are known (to me!) byproducts of the current implementation. If my crazy idea of adding opcodes to speed up f-strings flies, then this issue will go away. I consider this a corner case that doesn't need to be addressed before committing this

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: I’m actually trying out your patch now. A couple strange errors and observations: >>> f"{'{'}" # Why is this allowed in an outer format expression-- '{' >>> f"{3:{'{'}>10}" # --but not inside a format specifier? SyntaxError: nesting of '{' in format specifier

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Thanks again, Martin. I've found 4 bugs so far, based on your suggested tests. The ones I haven't fixed are: 'fur' strings don't work (something in the lexer), and triple quoted strings don't work correctly. I'm working on both of those, and should have an

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: It turns out 'fur' strings aren't a thing, because 'ur' strings aren't. >From tokenizer.c: /* ur"" and ru"" are not supported */ And the PEP: https://www.python.org/dev/peps/pep-0414/#exclusion-of-raw-unicode-literals I'll add a test to make sure this fails.

[issue24965] Implement PEP 498: Literal String Formatting

Martin Panter added the comment: Another version of that AST that is better for my digestion: f'a={a}' Module(body=[Expr( value=JoinedStr(values=[ Str(s='a='), FormattedValue( value=Name(id='a', ctx=Load()), conversion=0,

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Thanks, Martin. I've posted my replies. I'll add some more tests, and work on the triple quoted string bug. -- ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: I discussed it with Guido and added 'F' to the PEP. -- ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: This version does dynamic allocation for the expression list, and fixes some memory leaks and early decrefs. I think it's complete, but I'll take some more passes through it checking for leaks. -- ___ Python

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Eric V. Smith : Added file: http://bugs.python.org/file40430/pep-498-4.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: The good news is that the performance is pretty good, and finally I have a case where I can beat %-formatting: $ ./python.bat -mtimeit -s 'a=2' "'%s' % a" 100 loops, best of 3: 0.883 usec per loop $ ./python.bat -mtimeit -s 'a=2' '"{}".format(a)' 100

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Eric V. Smith : Removed file: http://bugs.python.org/file40302/pep-498.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Eric V. Smith : Removed file: http://bugs.python.org/file40316/pep-498-1.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Eric V. Smith : Removed file: http://bugs.python.org/file40317/pep-498-2.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: This implements the accepted PEP 498. The only other real change I plan on making is to do dynamic memory allocation when building the expressions that make up a JoinedStr AST node. The code has all of the places to do that already laid out, it's just a matter

[issue24965] Implement PEP 498: Literal String Formatting

Barry A. Warsaw added the comment: On Sep 09, 2015, at 11:57 PM, Eric V. Smith wrote: >There's one nit where I accept 'f' and 'F', but the PEP just says 'f'. I'm >not sure if we should accept the upper case version. I'd think not, but all >of the other ones (b, r, and u) do. I think it should

[issue24965] Implement PEP 498: Literal String Formatting

Roundup Robot added the comment: New changeset a0194ec4195c by Eric V. Smith in branch 'default': Removed Implementation Limitations section. While the version of the code on http://bugs.python.org/issue24965 has the 255 expression limitation, I'm going to remove this limit. The i18n section

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Fixed validate_exprs bug. -- Added file: http://bugs.python.org/file40316/pep-498-1.diff ___ Python tracker ___

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Yury Selivanov : -- nosy: +yselivanov ___ Python tracker ___ ___

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: Make sure f-strings are identified as literals in error messages. -- Added file: http://bugs.python.org/file40317/pep-498-2.diff ___ Python tracker

[issue24965] Implement PEP 498: Literal String Formatting

New submission from Eric V. Smith: See PEP 498. f'New for Python {sys.version.split()[0]}' 'New for Python 3.6.0a0' -- assignee: eric.smith components: Interpreter Core files: pep-498.diff keywords: patch messages: 249362 nosy: eric.smith priority: normal severity: normal status: open

[issue24965] Implement PEP 498: Literal String Formatting

Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24965 ___ ___ Python-bugs-list mailing

[issue24965] Implement PEP 498: Literal String Formatting

Eric V. Smith added the comment: One thing I've done in this implementation is to build up a string to pass to str.format(), instead of using the original string. This new string uses positional parameters instead of named parameters. I had originally proposed to add a string.interpolate() to