[issue28638] Optimize namedtuple creation

Raymond Hettinger added the comment: New changeset 8b57d7363916869357848e666d03fa7614c47897 by Raymond Hettinger in branch 'master': bpo-28638: Optimize namedtuple() creation time by minimizing use of exec() (#3454)

[issue28638] Optimize namedtuple creation

Changes by Raymond Hettinger : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue28638] Optimize namedtuple creation

Serhiy Storchaka added the comment: Microbenchmark for caching docstrings: $ ./python -m perf timeit -s "from collections import namedtuple; names = ['field%d' % i for i in range(1000)]" -- "namedtuple('A', names)" With sys.intern(): Mean +- std dev: 3.57 ms +- 0.05 ms With Python-level

[issue28638] Optimize namedtuple creation

Josh Rosenberg added the comment: Side-note: Some of the objections to a C level namedtuple implementation appear to be based on the maintenance hurdle, and other have noted that a structseq-based namedtuple might be an option. I have previously attempted to write a C replacement for

[issue28638] Optimize namedtuple creation

Changes by Raymond Hettinger : -- pull_requests: +3451 ___ Python tracker ___

[issue28638] Optimize namedtuple creation

Changes by Ethan Smith : -- nosy: +Ethan Smith ___ Python tracker ___ ___ Python-bugs-list

[issue28638] Optimize namedtuple creation

Raymond Hettinger added the comment: > it would be *nice* to not only optimization the creation > but also attribute access by name FWIW, once the property/itemgetter pair are instantiated in the NT class, the actual lookup runs through them at C speed (no pure python steps). There is not

[issue28638] Optimize namedtuple creation

Guido van Rossum added the comment: Yeah, it looks like the standard `_pickle` and `pickle` solution would work here. -- ___ Python tracker ___

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: General note about this issue: while the issie title is "Optimize namedtuple creation", it would be *nice* to not only optimization the creation but also attribute access by name: http://bugs.python.org/issue28638#msg298499 Maybe we can have a very fast C

[issue28638] Optimize namedtuple creation

INADA Naoki added the comment: I didn't say "let's not do it". I just want to focus on pure Python implementation at this issue, because this thread is too long already. Feel free to open new issue about C implementation. Even if C implementation is added later, pure Python optimization can

[issue28638] Optimize namedtuple creation

Giampaolo Rodola' added the comment: > While "40x faster" is more 10x faster than "4x faster", C > implementation can boost only CPython and makes maintenance more harder. As a counter argument against "let's not do it because it'll be harder to maintain" I'd like to point out that namedtuple

[issue28638] Optimize namedtuple creation

INADA Naoki added the comment: I want to focus on pure Python implementation in this issue. While "40x faster" is more 10x faster than "4x faster", C implementation can boost only CPython and makes maintenance more harder. And sometimes "more 10x faster" is not so important. For example, say

[issue28638] Optimize namedtuple creation

Jelle Zijlstra added the comment: Thanks Joe! I adapted your benchmark suite to also run my implementation. See https://github.com/JelleZijlstra/cnamedtuple/commit/61b6fbf4de37f8131ab43c619593327004974e52 for the code and results. The results are consistent with what we've seen before. Joe's

[issue28638] Optimize namedtuple creation

Jelle Zijlstra added the comment: I benchmarked some common namedtuple operations with the following script: #!/bin/bash echo 'namedtuple creation' ./python -m timeit -s 'from collections import namedtuple' 'x = namedtuple("x", ["a", "b", "c"])' echo 'namedtuple instantiation' ./python -m

[issue28638] Optimize namedtuple creation

Joe Jevnik added the comment: I added a benchmark suite (using Victor's perf utility) to cnamedtuple. The results are here: https://github.com/ll/cnamedtuple#benchmarks To summarize: type creation is much faster; instance creation and named attribute access are a bit faster.

[issue28638] Optimize namedtuple creation

Giampaolo Rodola' added the comment: > Should we consider a C-based implementation like > https://github.com/ll/cnamedtuple? > It could improve speed even more, but would be harder to maintain and > test and harder to keep compatible. My sense is that it's not worth > it unless

[issue28638] Optimize namedtuple creation

Changes by Antoine Pitrou : -- stage: resolved -> patch review ___ Python tracker ___ ___

[issue28638] Optimize namedtuple creation

Christoph Reiter added the comment: Why not just do the following: >>> from collections import namedtuple >>> Point = namedtuple('Point', ['x', 'y']) >>> Point._source "from collections import namedtuple\nPoint = namedtuple('Point', ['x', 'y'])\n" >>> The docs make it seems as if the primary

[issue28638] Optimize namedtuple creation

Guido van Rossum added the comment: Thanks Raymond and Jelle. The bar for a reimplementation in C is much higher (though we'll have to agree that Jelle's version is fast enough before we reject it). The bar for backporting this to 3.6 is much higher as well and I think it's not worth

[issue28638] Optimize namedtuple creation

Changes by Jelle Zijlstra : -- resolution: rejected -> ___ Python tracker ___ ___

[issue28638] Optimize namedtuple creation

Jelle Zijlstra added the comment: Should we consider a C-based implementation like https://github.com/ll/cnamedtuple? It could improve speed even more, but would be harder to maintain and test and harder to keep compatible. My sense is that it's not worth it unless benchmarks show a

[issue28638] Optimize namedtuple creation

Raymond Hettinger added the comment: Re-opening per discussion on python-dev. Goals: * Extend Jelle's patch to incorporate lazy support for "_source" and "verbose" so that the API is unchanged from the user's point of view. * Make sure the current test suite still passes and that the current

[issue28638] Optimize namedtuple creation

Guido van Rossum added the comment: On python-dev Raymond agreed to reopen the issue and consider Jelle's implementation (https://github.com/python/cpython/pull/2736). -- nosy: +gvanrossum ___ Python tracker

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: > It's possible to expose StructSeq somewhere. Hum, when I mentioned structseq: my idea was more to reimplement namedtuple using the existing structseq code, since structseq is well tested and very fast. -- ___

[issue28638] Optimize namedtuple creation

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___

[issue28638] Optimize namedtuple creation

INADA Naoki added the comment: I respect Raymond's rejection. But I want to write down why I like Jelle's approach. Currently, functools is the only module which is very popular. But leaving this means every new namedtuple makes startup time about 0.6ms slower. This is also problem for

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: > So structseq is 1.9x faster than namedtuple to get an attribute by name. Oops, I wrote it backward: So namedtuple is 1.9x slower than structseq to get an attribute by name. (1.9x slower doesn't mean 1.9x faster, sorry.) --

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: > Speed isn't everything, and it certainly isn't adequate justification for > breaking public APIs that have been around for years. What about the memory usage? > See my old issue #19640 (...) msg203271: """ I found this issue while using my tracemalloc

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: Benchmark comparing collections.namedtuple to structseq, to get an attribute: * Getting an attribute by name (obj.attr): Mean +- std dev: [name_structseq] 24.1 ns +- 0.5 ns -> [name_namedtuple] 45.7 ns +- 1.9 ns: 1.90x slower (+90%) * Getting an attribute by

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: Yes, I'm saying you need a really long justification to explain why you want to break backwards compatibility solely for a speed increase. For namedtuple instances, the leading underscore does *NOT* indicate a private attribute - it's just there to avoid

[issue28638] Optimize namedtuple creation

STINNER Victor added the comment: Sorry, I don't have much data at this point, but it's not the first time that I noticed that namedtuple is super slow. We have much more efficient code like structseq in C. Why not reusing it at least in our stdlib modules? About the _source attribute,

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: Check the issue history - the issue has been rejected by Raymond, and then reopened for further debate by other core developers multiple times. That's not a reasonable approach to requesting reconsideration of a module/API maintainers design decision. I

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: There's a path for escalation when you disagree with the decision of a module/API maintainer (in this case, Raymond): bringing the issue closure up on python-dev for wider discussion. It *isn't* repeatedly reopening the issue after they have already made their

[issue28638] Optimize namedtuple creation

Antoine Pitrou added the comment: Just because I disagree with you doesn't mean I'm pestering anyone. Can you stop being so obnoxious? -- ___ Python tracker

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: So unless and until he gets overruled by Guido, Raymond's decision to reject the proposed change stands. -- resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker

[issue28638] Optimize namedtuple creation

Antoine Pitrou added the comment: Nick, can you stop closing an issue where the discussion hasn't been settled? This isn't civil. -- resolution: rejected -> stage: resolved -> status: closed -> open ___ Python tracker

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: Folks, you're talking about removing a *public*, *documented* API from the standard library. The onus would thus be on you to prove *lack* of use, *and* provide adequate justification for the compatibility break, not on anyone else to prove that it's

[issue28638] Optimize namedtuple creation

Antoine Pitrou added the comment: I disagree with the rejection of this request. The idea that "_source is an essential feature" should be backed by usage statistics instead of being hand-waved as rejection cause. -- nosy: +pitrou resolution: rejected -> stage: resolved -> status:

[issue28638] Optimize namedtuple creation

Changes by Raymond Hettinger : -- resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker ___

[issue28638] Optimize namedtuple creation

Nick Coghlan added the comment: I agree with Raymond here - the standard library's startup benchmarks are *NOT* normal code execution paths, since normal code execution is dominated by the actual operation being performed, and hence startup micro-optimizations vanish into the noise.

[issue28638] Optimize namedtuple creation

Raymond Hettinger added the comment: > creates only one necessary backwards compatibility break > (we no longer have _source). IMO, this is an essential feature. It allows people to easily build their own variants, to divorce the generated code from the generator, and to fully understand

[issue28638] Optimize namedtuple creation

Changes by Jelle Zijlstra : -- pull_requests: +2796 ___ Python tracker ___ ___

[issue28638] Optimize namedtuple creation

INADA Naoki added the comment: I like your idea. Would you make pull request? -- resolution: rejected -> status: closed -> open title: Creating namedtuple is too slow to be used in common stdlib (e.g. functools) -> Optimize namedtuple creation ___