[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: Now that I've looked at PyUnicode_New, I agree with using PyUnicode_New(0, 0). I can't imagine we could measure the difference with optimizing it in the opcode itself before calling PyUnicode_New. Thanks for adding this, Serhiy. I think it's great stuff, and

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: Thank you for your review Eric. I considered passing NULL, but as Antti said, it is already used for space separated concatenation. PyUnicode_New(0, 0) is the obvious way to get an empty string, and I think it is fast enough. If this affects performance we

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Roundup Robot added the comment: New changeset 28e280915508 by Serhiy Storchaka in branch 'default': Issue #27078: Added BUILD_STRING opcode. Optimized f-strings evaluation. https://hg.python.org/cpython/rev/28e280915508 -- nosy: +python-dev ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: Though it is clean to do like this: let _PyUnicode_JoinArray have `NULL` mean empty string, as it is more logical anyway; then PyUnicode_Join itself just needs to: if (separator == NULL) { separator = PyUnicode_FromOrdinal(' '); /* check

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: Of course. Never mind. LGTM. -- ___ Python tracker ___ ___ Python-bugs-list

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: I tried to post a comment on rietveld, but 500 infernal error... PyUnicode_New(0, 0) will return unicode_empty. If unicode_empty is NULL, then it will also initialize it. It would be cleanest if unicode_empty was statically created. NULL cannot be used as the

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: Left a review comment. I'd like to see this in before 3.6 beta 1. -- ___ Python tracker ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Raymond Hettinger added the comment: I put this in the "new feature" category in the sense that after beta we're trying to stabilize the code base. Introducing a new opcode is a higher risk change that needs to go in the release cycle as early as possible. --

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Demur Rumed added the comment: I don't think lack of precedence is a reason to say new opcodes are new features. More that generally new opcodes are created for new features -- ___ Python tracker

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: So does this (new opcode) count as a new feature? It would be great to give f'' strings a flying start, saying that not only they're cool, they're also faster than anything that you've used before. Here some more mini-benchmarks with serhiy's patch2 applied,

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: Since this introduces a new opcode, this is a new feature. Seems opcodes never were added at beta stage before 3.5b1. -- ___ Python tracker

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: I intend to review this. As this is not a new feature, it doesn't need to be completed by beta 1. I'm focusing my energies on new features, then I'll look at this. -- ___ Python tracker

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: I would very much like to see this in 3.6. Who could review it? -- ___ Python tracker ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: Nice idea, Antti. But I tried to implement it, and surprisingly found that this approach is slower than FORMAT_VALUE + BUILD_STRING. At least for this particular example. Perhaps because we can't use a stack and need to allocate a new tuple containing

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: Serhiy suggested this in Rietveld: > For additional optimization we can pack all constant strings, parsed formats > and > flags in one constant object and use the single LOAD_CONST. But this requires > much larger changes (perhaps including changing the marshal

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: Thanks Serhiy, I was writing my comment for a long time, and only now noticed that you'd already posted the patch. Indeed, it seems that not only is this the fastest method, it might also be the fastest string concatenation method in the history of Python. I

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Demur Rumed : -- nosy: +rhettinger ___ Python tracker ___ ___ Python-bugs-list

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: It seems Eric has done some special casing for strings already in FORMAT_VALUE. Here are the results from my computer after applying Demur's patch for concatenating *strings*. python3.6 -m timeit -s "x = 'a'" -- '"X is %s" % x' 100 loops, best of

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: Proposed patch adds the BUILD_STRING opcode and speeds up PyObject_Format() in common cases. It makes f-strings the fastest method for simple formatting. $ ./python -m timeit -s "x = 2" -- 'f"X is {x}"' 100 loops, best of 3: 0.347 usec per loop

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: Yet one case for comparison (with msg270095): $ ./python -m timeit -s "x = 2" -- 'f"X is {x!s}"' 100 loops, best of 3: 0.625 usec per loop Seems f'{x!s}' is the fastest way to stringify a value. Thus there is an opportunity to speed up default

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Demur Rumed : -- type: enhancement -> performance ___ Python tracker ___ ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Demur Rumed added the comment: I'm not understanding your message. We don't call FORMAT_VALUE on constant strings in f"x is {x}" & FORMAT_VALUE doesn't take an argument. Are you saying in a hypothetical FORMAT_VALUE where BUILD_STRING takes a set of objects & applies formatting to them, thus

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: Yet the test cases just prove what is so expensive there: name lookups (global name `str`; looking up `join` on a string instance); building a tuple (for function arguments) is expensive as well. Of course `__format__` will be costly as well as it is not a

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Demur Rumed added the comment: fstrtup2.patch is a bit more of an involved optimization for when we are joining 2 strings. Instead it emits BINARY_ADD. This may be considered too 'niche' since it only triggers when the substitution occurs at the start or end of a string & there is only one

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Demur Rumed added the comment: Benchmarked f'X is {x}' with BUILD_TUPLE change: Before: 6.62 usec per loop After: 6.33 usec per loop f'X is {x} {x+2} {x+3}' Before: 15.1 usec per loop After: 14.7 usec per loop Attached patch -- keywords: +patch Added file:

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Demur Rumed added the comment: The simplest perf fix is to first use BUILD_TUPLE instead of BUILD_LIST timeit 'x=1;(x,x,x)' 0.36 usec per loop timeit 'x=1;[x,x,x]' 0.425 usec per loop Introducing a new opcode BUILD_STRING to inline PyTuple_New + PyUnicode_Join to replace BUILD_TUPLE +

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: Ah so it seems. Somehow I thought __format__ was slotted, but that is not the case and it needs to be looked up, and what is worse, of course a tuple needs to be built as well. Oh well, at least it should be quite minimal to make it be faster than `f(x)`

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: > And the expected performance for optimal `f'X is {x}'` code would > be *faster* than `"'X is %s' % (x,)"` which still needs to > interpret the string at runtime, and build a proper tuple object > on stack. That's not necessarily true. The f-string version

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: And the expected performance for optimal `f'X is {x}'` code would be *faster* than `"'X is %s' % (x,)"` which still needs to interpret the string at runtime, and build a proper tuple object on stack. -- ___ Python

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Serhiy Storchaka added the comment: For comparison: $ ./python -m timeit -s "x = 2" -- "f'X is {x}'" 100 loops, best of 3: 1.04 usec per loop $ ./python -m timeit -s "x = 2; j = ''.join" -- "j(['X is ', f'{x}'])" 100 loops, best of 3: 0.93 usec per loop $ ./python -m timeit -s "x = 2"

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Serhiy Storchaka : -- nosy: +Demur Rumed ___ Python tracker ___ ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Antti Haapala added the comment: I am not an expert on writing new opcodes to CPython (having never done it, don't know where to change the disassembler and such, how to make compiler generate them properly and such), but I'd be glad to help with testing, timing and writing the possible

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Martijn Pieters added the comment: The catalyst for this question was a Stack Overflow question I answered: https://stackoverflow.com/questions/37365311/why-are-python-3-6-literal-formatted-strings-so-slow Compared the `str.format()` the BUILD_LIST is the bottleneck here; dropping the

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Martijn Pieters : -- nosy: +mjpieters ___ Python tracker ___ ___ Python-bugs-list

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Eric V. Smith added the comment: I considered doing this, and have some of it implemented. I discussed it with Larry Hastings and Mark Shannon at PyCon Ireland, and we decided to just use ''.format() and the new FORMAT_VALUE opcode, since that was the simplest way to fix the previous

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Eric V. Smith : -- assignee: -> eric.smith ___ Python tracker ___ ___

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

Changes by Ned Deily : -- nosy: +eric.smith ___ Python tracker ___ ___ Python-bugs-list

[issue27078] Make f'' strings faster than .format: BUILD_STRING opcode?

New submission from Antti Haapala: I benchmarked some f'' strings against .format, and surprisingly f'' was slower than .format in about all the simple cases I could think of. I guess this is because f'' strings implicitly use `''.join([])`. The byte code for f'foo is {foo}' currently is 1