[issue47175] Allow applications to tune the condition that triggers a GIL release and implementation choice in hashlib
New submission from Gregory P. Smith : ## Background All `hashlib` computations and `binascii.crc32` and `zlib.crc32` release the GIL around their computational core. But they use a hard coded length check to determine when to do so, or always do it. That already accomplishes the larger good of releasing the GIL on big computations. But _probably_ just serves to slow down smaller ones when a GIL release adds more overhead than a context switch to another thread could meaningfully provide in terms of forward progress. ## Desire 1 Determine if a threshold should exist at all (should we just always release the GIL for these?) and if so, allow it to be tuned on a per algorithm basis. This comes at the same time as other enhancements like bpo-47102 and its windows and macos cousins could shift us towards using OS kernel APIs for a subset of algorithms where available - which may effectively "always release" the GIL on OS APIs that are virtual IO call based such as bpo-47102's. ## Desire 2 When multiple implementations of an algorithm may be available, allow the user to configure data length thresholds for when each one is triggered. Without meaningfully slowing most things down by adding such logic. Different implementations have different setup and finalization time which can make them more useful for large data rather than tiny. --- I'm marking this low priority as it veers towards over-optimization. :) Creating benchmarks and a thing to live in Tools/ that people could run on their target platform to provide a tuning suggestion of what thresholds work best for their needs. Related inspiring work: OSes often benchmark several algorithm implementations up front to pick a "best" to use for a given platform (ex: see what the Linux kernel does for hashes and raid algorithms). This extends that idea and acknowledges latency as important, not exclusively thru-put. -- messages: 416401 nosy: gregory.p.smith priority: low severity: normal stage: needs patch status: open title: Allow applications to tune the condition that triggers a GIL release and implementation choice in hashlib type: enhancement ___ Python tracker <https://bugs.python.org/issue47175> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19160] Inconsistent size for GIL release in hashlib
New submission from Jesús Cea Avión: In http://docs.python.org/3.3/library/hashlib.html there is a contradiction: For better multithreading performance, the Python GIL is released for strings of more than 2047 bytes at object creation or on update. [...] Changed in version 3.1: The Python GIL is released to allow other threads to run while hash updates on data larger than 2048 bytes is taking place when using hash algorithms supplied by OpenSSL. So is it 2047 or 2048? :) -- assignee: docs@python components: Documentation keywords: easy messages: 198930 nosy: docs@python, jcea priority: normal severity: normal status: open title: Inconsistent size for GIL release in hashlib versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19160 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19160] Inconsistent size for GIL release in hashlib
Berker Peksag added the comment: It looks like it is 2048: http://hg.python.org/cpython/rev/f9f5d9047a05/ See also issue 4751. -- nosy: +berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19160 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19160] Inconsistent size for GIL release in hashlib
Jesús Cea Avión added the comment:
For extra safety, I just checked the source of 3.4:
#define HASHLIB_GIL_MINSIZE 2048
[...]
if (self-lock == NULL view.len = HASHLIB_GIL_MINSIZE) {
[...]
if (view.len = HASHLIB_GIL_MINSIZE) {
[...]
if (len = HASHLIB_GIL_MINSIZE) {
So, yes, the GIL is released if len = 2048 bytes.
BTW, in Python 3.x hashes can't be calculated on strings, but bytes.
I take care of this. Commit in 5 minutes.
--
assignee: docs@python - jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19160
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19160] Inconsistent size for GIL release in hashlib
Roundup Robot added the comment: New changeset 310c26df3234 by Jesus Cea in branch '3.3': Close #19160: Inconsistent size for GIL release in hashlib http://hg.python.org/cpython/rev/310c26df3234 New changeset 9503097ce1b7 by Jesus Cea in branch 'default': MERGE: Close #19160: Inconsistent size for GIL release in hashlib http://hg.python.org/cpython/rev/9503097ce1b7 -- nosy: +python-dev resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19160 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19160] Inconsistent size for GIL release in hashlib
Roundup Robot added the comment: New changeset bfebfadfc4aa by Jesus Cea in branch '3.3': Close #19160: Inconsistent size for GIL release in hashlib http://hg.python.org/cpython/rev/bfebfadfc4aa New changeset 989ea05b2500 by Jesus Cea in branch 'default': MERGE: Close #19160: Inconsistent size for GIL release in hashlib http://hg.python.org/cpython/rev/989ea05b2500 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19160 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
GIL release
Hi, Does anyone know whether there is a way for a python thread to release the global interpreter lock, and let all other threads have a chance at running before re-acquiring it? Does the thread scheduling follow a round-robin method? Thanks, agb. -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL release
Alastair Basden wrote: Hi, Does anyone know whether there is a way for a python thread to release the global interpreter lock, and let all other threads have a chance at running before re-acquiring it? Does the thread scheduling follow a round-robin method? The thread itself are scheduled by the OS ... the access to the GIL is managed by 'cooperative scheduling'. --Armin Thanks, agb. -- http://mail.python.org/mailman/listinfo/python-list
Re: GIL release
Do you mean in Python or in C ? In C this is described in details in the documentation. In Python, I don't think there is a way ! If you want to do so you'll want to use a micro sleep ... Pierre Alastair Basden a écrit : Hi, Does anyone know whether there is a way for a python thread to release the global interpreter lock, and let all other threads have a chance at running before re-acquiring it? Does the thread scheduling follow a round-robin method? Thanks, agb. -- http://mail.python.org/mailman/listinfo/python-list
