[Python-Dev] the new(-ish) dict ordering vs hash randomization
I'm sure we've already had this conversation, but my google-fu is failing me. Can someone provide a link to a discussion explaining why the new ordering of dictionaries does not defeat the hash-randomization non-ordering we added a few versions ago? -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
Hi, I tried two options to add support for FASTCALL on calling an object: add a flag in tp_flags and reuse tp_call, or add a new tp_fastcall slot. I failed to implement correctly any of these two options. There are multiple issues with tp_fastcall: * ABI issue: it's possible to load a C extension using the old ABI, without tp_fastcall: it's not possible to write type->tp_fastcall on such type. This limitation causes different issues. * If tp_call is modified, tp_fastcall may be outdated. Same if tp_fastcall is modified. What happens on "del obj.__call__" or "del type.__call__"? * Many public functions of the C API still requires the tuple and dict to pass positional and keyword arguments, so a compatibility layer is required to types who only want to implement FASTCALL. Related issue: what is something calls tp_call with (args: tuple, kwargs: dict)? Crash or call a compatibility layer converting arguments to FASTCALL calling convention? Reusing tp_call for FASTCALL cause similar or worse issues. I abandoned my idea for two reasons: 1) in the worst case, my changes caused a crash which is not accepted for an optimization. My first intent was to removed the property_descr_get() hack because its implementation is fragile and caused crashes. 2) we implemented a lot of other optimizations which made calls faster without having to touch tp_call nor tp_fastcall. The benefit of FASTCALL for tp_call/tp_fastcall was not really significant. Victor 2018-06-18 7:55 GMT+02:00 Jeroen Demeyer : > On 2018-06-18 03:34, INADA Naoki wrote: >> >> Victor had tried to add `tp_fastcall` slot, but he suspended his effort >> because >> it's benefit is not enough for it's complexity. >> https://bugs.python.org/issue29259 > > > I has a quick look at that patch and it's really orthogonal to what I'm > proposing. I'm proposing to use the slot *instead* of existing fastcall > optimizations. Victor's patch was about adding fastcall support to classes > that didn't support it before. > > > > Jeroen. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] the new(-ish) dict ordering vs hash randomization
On Mon, 18 Jun 2018 06:13:15 -0700 Ethan Furman wrote: > I'm sure we've already had this conversation, but my google-fu is failing me. > > Can someone provide a link to a discussion explaining why the new ordering of > dictionaries does not defeat the > hash-randomization non-ordering we added a few versions ago? Because the aim of hash randomization was not to make iteration order unpredictable, it was to make hash collisions unpredictable. The solution used to make hash collisions unpredictable was to make hash values themselves unpredictable, and that had the side effect of also making iteration order unpredictable. But the new dict implementation is able to provide a deterministic iteration order even with non-deterministic hash values. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] the new(-ish) dict ordering vs hash randomization
On 06/18/18 15:13, Ethan Furman wrote: I'm sure we've already had this conversation, but my google-fu is failing me. Can someone provide a link to a discussion explaining why the new ordering of dictionaries does not defeat the hash-randomization non-ordering we added a few versions ago? Hi, Modern dicts have an array of contents (which is used for iterating the dict, and thus iteration doesn't touch hashes at all), and a separate hash table of indexes (which still enjoys the benefits of hash randomization). See Raymond Hettinger's initial post from 2012: https://mail.python.org/pipermail/python-dev/2012-December/123028.html A technical overview of the idea is on the PyPy blog: https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
On 2018-06-18 15:09, Victor Stinner wrote: 2) we implemented a lot of other optimizations which made calls faster without having to touch tp_call nor tp_fastcall. And that's a problem because these optimizations typically only work for specific classes. My PEP wants to replace those by something more structural. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
On Mon, Jun 18, 2018 at 11:33 PM Jeroen Demeyer wrote: > On 2018-06-18 15:09, Victor Stinner wrote: > > 2) we implemented a lot of other optimizations which made calls faster > > without having to touch tp_call nor tp_fastcall. > > And that's a problem because these optimizations typically only work for > specific classes. My PEP wants to replace those by something more > structural. > And we need data how much it speedup some applications, not only microbenchmarks. Speeding up most python function and some bultin functions was very significant. But I doubt making some 3rd party call 20% faster can make real applications significant faster. -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
Like Inada-san, I would like to see the implementation first. On Mon, Jun 18, 2018, 07:33 Jeroen Demeyer wrote: > On 2018-06-18 15:09, Victor Stinner wrote: > > 2) we implemented a lot of other optimizations which made calls faster > > without having to touch tp_call nor tp_fastcall. > > And that's a problem because these optimizations typically only work for > specific classes. My PEP wants to replace those by something more > structural. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 2018 Python Language Summit coverage, part 2
Hola python-dev, Here is some more coverage from the Python Language Summit. As usual, I am posting SubscriberLinks for articles that are still behind the paywall. LWN subscribers can always see our content right away; one week after they are published in a weekly edition, they become freely available for everyone. SubscriberLinks are a way around the paywall. Please feel free to share the SubscriberLinks I am posting here. The starting point is here: https://lwn.net/Articles/754152/ That is an overview article with links to the individual articles. It will be updated as I add more articles. Only 3 more to go after this. Here is what was added since my previous post (which is here: https://lwn.net/ml/python-dev/20180606155653.264c9566%40gallinule/ ) Linux distributions and Python 2 - https://lwn.net/SubscriberLink/756628/7a85f7b28ae3f690/ A Python static typing update - https://lwn.net/SubscriberLink/757218/6f47fe0675cbaf01/ Python virtual environments - https://lwn.net/SubscriberLink/757354/fd82c236dff2de13/ I will post again with the last three, which should be later this week ... enjoy! jake -- Jake Edge - LWN - [email protected] - http://lwn.net ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
Victor Stinner schrieb am 18.06.2018 um 15:09: > I tried two options to add support for FASTCALL on calling an object: > add a flag in tp_flags and reuse tp_call, or add a new tp_fastcall > slot. I failed to implement correctly any of these two options. > > There are multiple issues with tp_fastcall: > > * ABI issue: it's possible to load a C extension using the old ABI, > without tp_fastcall: it's not possible to write type->tp_fastcall on > such type. This limitation causes different issues. Not a problem if we rededicate the unused (since Py3.0) "tp_print" slot for it. Even better, since the slot exists already in Py3.0+, tools like Cython, NumPy (with its ufuncs etc.) or generic function dispatchers, basically anything that benefits from fast calls, can enable support for it in all CPython 3.x versions and benefit from faster calls among each other, independent of the support in CPython. The explicit type flag opt-in that the PEP proposes makes this completely safe. > * If tp_call is modified, tp_fastcall may be outdated. Same if > tp_fastcall is modified. Slots are fixed at type creation and should never be modified afterwards. > What happens on "del obj.__call__" or "del type.__call__"? $ python3.7 -c 'del len.__call__' Traceback (most recent call last): File "", line 1, in AttributeError: 'builtin_function_or_method' object attribute '__call__' is read-only $ python3.7 -c 'del type.__call__' Traceback (most recent call last): File "", line 1, in TypeError: can't set attributes of built-in/extension type 'type' And a really lovely one: $ python3.7 -c 'del (lambda:0).__call__' Traceback (most recent call last): File "", line 1, in AttributeError: __call__ > * Many public functions of the C API still requires the tuple and dict > to pass positional and keyword arguments, so a compatibility layer is > required to types who only want to implement FASTCALL. Well, yes. It would require a trivial piece of code to map between the two. Fine with me. > Related issue: > what is something calls tp_call with (args: tuple, kwargs: dict)? > Crash or call a compatibility layer converting arguments to FASTCALL > calling convention? The latter, obviously. Also easy to implement, with the usual undefined dict order caveat (although that's probably solved when running in Py3.6+). > I abandoned my idea for two reasons: > > 1) in the worst case, my changes caused a crash which is not accepted > for an optimization. This isn't really an optimisation. It's a generalisation of the call protocol. > My first intent was to removed the > property_descr_get() hack because its implementation is fragile and > caused crashes. Not sure which hack you mean. > 2) we implemented a lot of other optimizations which made calls faster > without having to touch tp_call nor tp_fastcall. The benefit of > FASTCALL for tp_call/tp_fastcall was not really significant. What Jeroen said. Cleaning up the implementation and generalising the call protocol is going to open up a wonderfully bright future for CPython. :) Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev)
On 06/08/2018 12:48 AM, Victor Stinner wrote: Question: Do you think that bugs spotted by a GC collection are common enough to change the GC thresholds in development mode (new -X dev flag of Python 3.7)? I'd prefer that the development / debug environment be as much like production use as possible, so that surprises crop up during development rather than after deployment. Additional monitoring is fine, but I think changing behavior is a no-no. Cheers, //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
On 2018-06-18 16:55, INADA Naoki wrote: Speeding up most python function and some bultin functions was very significant. But I doubt making some 3rd party call 20% faster can make real applications significant faster. These two sentences are almost contradictory. I find it strange to claim that a given optimization was "very significant" in specific cases while saying that the same optimization won't matter in other cases. People *have* done benchmarks for actual code and this is causing actual slow-downs of around 20% in actual applications. That is the main reason why I am trying to push this PEP (or PEP 575 which solves the same problem in a different way). Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 575 (Unifying function/method classes) update
On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer wrote: > On 2018-06-18 16:55, INADA Naoki wrote: > > Speeding up most python function and some bultin functions was very > > significant. > > But I doubt making some 3rd party call 20% faster can make real > > applications significant faster. > > These two sentences are almost contradictory. I find it strange to claim > that a given optimization was "very significant" in specific cases while > saying that the same optimization won't matter in other cases. > It's not contradictory because there is basis: In most real world Python application, number of calling Python methods or bulitin functions are much more than other calls. For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL was rejected because it's implementation is complex and it's performance gain is not significant enough on macro benchmarks. And I doubt number of 3rd party calls are much more than calling builtin tp_init or tp_new. Of course, current benchmark suite [1] doesn't cover all types of real world Python application. You can create pull request which add benchmark for real world application which depends on massive 3rd party calls. [1] https://github.com/python/performance Regards, -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
