[Python-Dev] Request for review: binary op dispatch rules for subclasses
Over a year ago, I made a pull request ( https://github.com/python/cpython/pull/1325) to fix a long-standing issue with how Python handles dispatch for arithmetic binary operations involving subclasses (https://bugs.python.org/issue30140). I pinged the bug several times, but I'm still waiting for a review, which would be greatly appreciated! Best, Stephan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On 2018-09-14, Larry Hastings wrote: > [..] adding the stat calls back in costs you half the startup. So > any mechanism where we're talking to the disk _at all_ simply > isn't going to be as fast. Okay, so if we use hundreds of small .pyc files scattered all over the disk, that's bad? Who would have thunk it. ;-P We could have a new format, .pya (compiled python archive) that has data for many .pyc files in it. In normal runs you would have one or just and handlful of these things (e.g. one for stdlib, one for your app and all the packages it uses). Then you mmap these just once and rely on OS page faults to bring in the data as you need it. The .pya would have a hash table at the start or end that tells you the offset for each module. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On 09/14/2018 02:54 PM, Neil Schemenauer wrote: On 2018-09-14, Larry Hastings wrote: [...] improvement 0.21242667903482038 % I assume that should be 21.2 % othewise I recommend you abandon the idea. ;-P Yeah, that thing you said. I wonder how much of the speedup relies on putting it in the data segment (i.e. using linker/loader to essentially handle the unmarshal). What if you had a new marshal format that only needed a light 2nd pass in order to fix up the data loaded from disk? Some experimentation would be in order. I can suggest that, based on conversation from Carl, that adding the stat calls back in costs you half the startup. So any mechanism where we're talking to the disk _at all_ simply isn't going to be as fast. //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
On 2018-09-14, Larry Hastings wrote: [...] > improvement 0.21242667903482038 % I assume that should be 21.2 % othewise I recommend you abandon the idea. ;-P > The downside of the patch: for these modules it ignores the Python files on > disk--it doesn't even stat them. Having a command-line/env var to turn this on/off would be an acceptable fix, IMHO. If I'm running Python a server, I don't need to be editing .py modules and have them be recognized. Maybe have it turned off by default, at least at first. > Is it worth working on? I wonder how much of the speedup relies on putting it in the data segment (i.e. using linker/loader to essentially handle the unmarshal). What if you had a new marshal format that only needed a light 2nd pass in order to fix up the data loaded from disk? Yuri suggested looking at formats like Cap'n Proto. If the cost of the 2nd pass was not bad, you wouldn't have to rely on the platform C toolchain. Instead we can write .pyc files that hold this data. Then the speedup can work on all compiled Python modules, not just the ones you go through the special process that links them into the data segment. I suppose that might mean that .pyc files become arch specific. Maybe that's okay. As you said last night, there doesn't seem to be much low hanging fruit around anymore. So, 21% looks pretty decent. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
What follows is the text of issue 34690: https://bugs.python.org/issue34690 The PR is here: https://github.com/python/cpython/pull/9320 I don't know if we should be discussing this here on python-dev, or on bpo, or on Zulip, or on the soon-to-be-created Discourse. But maybe we can talk about it somewhere! //arry/ This patch was sent to me privately by Jeethu Rao at Facebook. It's a change they're working with internally to improve startup time. What I've been told by Carl Shapiro at Facebook is that we have their blessing to post it publicly / merge it / build upon it for CPython. Their patch was written for 3.6, I have massaged it to the point where it minimally works with 3.8. What the patch does: it takes all the Python modules that are loaded as part of interpreter startup and deserializes the marshalled .pyc file into precreated objects stored as static C data. You add this .C file to the Python build. Then there's a patch to Python itself (about 250 lines iirc) that teaches it to load modules from these data structures. I wrote a quick dumb test harness to compare this patch vs 3.8 stock. It runs a command line 500 times and uses time.perf_counter to time the process. On a fast quiescent laptop I observe a 21-22% improvement: cmdline: ['./python', '-c', 'pass'] 500 runs: sm38 average time 0.006302303705982922 best 0.006055746000129147 worst 0.0081656558437 clean38 average time 0.007969956444008858 best 0.007829047999621253 worst 0.00881221542109 improvement 0.20924239043734505 % cmdline: ['./python', '-c', 'import io'] 500 runs: sm38 average time 0.006297688038004708 best 0.00598076593309 worst 0.0072462130010535475 clean38 average time 0.007996319670004595 best 0.0078091849991324125 worst 0.009175700999549008 improvement 0.21242667903482038 % The downside of the patch: for these modules it ignores the Python files on disk--it doesn't even stat them. If you add stat calls you lose half of the speed improvement. I believe they added a work-around, where you can set a flag (command-line? environment variable? I don't know, I didn't go looking for it) that tells Python "don't use the frozen modules" and it loads all those files from disk. I don't propose to merge the patch in its current state. I think it would need a lot of work both in terms of "doing things the way Python does it" as well as just code smell (the serializer is implemented in both C and Python and jumps back and forth, also the build process for the serialized modules is pretty tiresome). Is it worth working on? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-09-07 - 2018-09-14) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6819 (-22) closed 39612 (+95) total 46431 (+73) Open issues with patches: 2716 Issues opened (38) == #23354: Loading 2 GiLOC file which raises exception causes wrong trace https://bugs.python.org/issue23354 reopened by benjamin.peterson #26544: platform.libc_ver() returns incorrect version number https://bugs.python.org/issue26544 reopened by vstinner #33083: math.factorial accepts non-integral Decimal instances https://bugs.python.org/issue33083 reopened by vstinner #34609: Idle Unitest https://bugs.python.org/issue34609 opened by pisc...@yahoo.fr #34610: Incorrect iteration of Manager.dict() method of the multiproce https://bugs.python.org/issue34610 opened by deltaclock #34613: asyncio.StreamReader initialization documentation incorrectly https://bugs.python.org/issue34613 opened by psycojoker #34616: implement "Async exec" https://bugs.python.org/issue34616 opened by mbussonn #34617: socket.recvfrom(): docs should warn about packet truncation wh https://bugs.python.org/issue34617 opened by Anees Ahmed #34620: Octal byte literals with a decimal value > 255 are silently tr https://bugs.python.org/issue34620 opened by bup #34623: _elementtree.c doesn't call XML_SetHashSalt() https://bugs.python.org/issue34623 opened by christian.heimes #34624: -W option does not accept module regexes https://bugs.python.org/issue34624 opened by coldfix #34626: PEP 384's PyType_Spec and PyType_Slot are not documented https://bugs.python.org/issue34626 opened by petr.viktorin #34628: urllib.request.urlopen fails when userinfo is present in URL https://bugs.python.org/issue34628 opened by ytvwld #34629: Python3 regression for urllib(2).urlopen(...).fp for chunked h https://bugs.python.org/issue34629 opened by tkruse #34630: Don't log ssl cert errors in asyncio https://bugs.python.org/issue34630 opened by asvetlov #34631: Upgrade to OpenSSL 1.1.1 https://bugs.python.org/issue34631 opened by christian.heimes #34632: Port importlib_metadata to Python 3.8 https://bugs.python.org/issue34632 opened by barry #34634: New asyncio streams API https://bugs.python.org/issue34634 opened by asvetlov #34639: PYTHONCOERCECLOCALE is ignored when using -E or -I option https://bugs.python.org/issue34639 opened by vstinner #34643: How to build Release Version of Python in Windows? https://bugs.python.org/issue34643 opened by Valentin Zhao #34648: Confirm the types of parameters of traceback.format_list and t https://bugs.python.org/issue34648 opened by Nathaniel Manista #34651: Disallow fork in a subinterpreter. https://bugs.python.org/issue34651 opened by eric.snow #34655: Support sendfile in asyncio streams API https://bugs.python.org/issue34655 opened by asvetlov #34656: memory exhaustion in Modules/_pickle.c:1393 https://bugs.python.org/issue34656 opened by shuoz #34659: Inconsistency between functools.reduce & itertools.accumulate https://bugs.python.org/issue34659 opened by lycantropos #34662: tarfile.TarFile may write corrupt files if not closed https://bugs.python.org/issue34662 opened by lordmauve #34663: Support POSIX_SPAWN_USEVFORK flag in posix_spawn https://bugs.python.org/issue34663 opened by pablogsal #34665: Py_FinalizeEx() - Bugs & caveats - Add info that NumPy and Pan https://bugs.python.org/issue34665 opened by jcmuel #34667: Review documentation section by section https://bugs.python.org/issue34667 opened by willingc #34669: test_ssl fails if SSLv2 is enabled https://bugs.python.org/issue34669 opened by benjamin.peterson #34670: Add set_post_handshake_auth for TLS 1.3 https://bugs.python.org/issue34670 opened by christian.heimes #34672: '%Z' strftime specifier never works with musl https://bugs.python.org/issue34672 opened by benjamin.peterson #34673: make the eval loop more editable https://bugs.python.org/issue34673 opened by benjamin.peterson #34676: Guarantie that divmod() and PyNumber_Divmod() return a 2-tuple https://bugs.python.org/issue34676 opened by serhiy.storchaka #34677: Event scheduler page example https://bugs.python.org/issue34677 opened by mediator42 #34678: need to remove the term "white space" https://bugs.python.org/issue34678 opened by dwaygig #34679: asyncio.add_signal_handler call fails if not on main thread https://bugs.python.org/issue34679 opened by jnwatson #34680: asyncio event_loop close fails off main thread if signal handl https://bugs.python.org/issue34680 opened by jnwatson Most recent 15 issues with no replies (15) == #34680: asyncio event_loop close fails off main thread if signal handl https://bugs.python.org/issue34680 #34679: asyncio.add_signal_handler call fails if not on main thread https://bugs.python.org/issue34
Re: [Python-Dev] Heap-allocated StructSequences
On 09/13/18 23:34, Neil Schemenauer wrote: On 2018-09-04, Eddie Elizondo wrote: Solution: * Fix the implementation of PyStructSequence_NewType: The best solution would be to fix the implementation of this function. This can easily be done by dynamically creating a PyType_Spec and calling PyType_FromSpec Hello Eddie, Thank you for spending time to look into this. Without studying the details of your patch, your approach sounds correct to me. I think we should be allocating types from the heap and use PyType_FromSpec. Having static type definitions living in the data segment cause too many issues. We have to assess how 3rd party extension modules would be affected by this change. Unless it is too hard to do, they should still compile (perhaps with warnings) after your fix. Do you know if that's the case? Looking at your changes to structseq.c, I can't tell easily. In any case, this should go into Victor's pythoncapi fork. That fork includes all the C-API cleanup we are hoping to make to CPython (assuming we can figure out the backwards and forwards compatibility issues). Nope, Victor's fork doesn't include all C-API cleanup. There's an older long-term effort (PEP-384, PEP-489, the current contenders 576/579/580, and PEP-573 for the future). Converting things to use PyType_FromSpec falls in there. As long as the old API still works, these changes should go in (but they might need a PEP). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
