[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by Antoine Pitrou : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: New changeset f3e6eadbcf4f3e0fe53f4784485b1c8464c7d282 by Antoine Pitrou (Miss Islington (bot)) in branch '3.6': [3.6] bpo-33021: Release the GIL during fstat() calls (GH-6019) (GH-6160)

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: New changeset 56cce1ca84344f519f7ed01024434c083031d354 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7': [3.7] bpo-33021: Release the GIL during fstat() calls (GH-6019) (GH-6159)

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by miss-islington : -- pull_requests: +5917 ___ Python tracker ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by miss-islington : -- pull_requests: +5916 stage: resolved -> patch review ___ Python tracker ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: Reopening for 3.x backports, as per Robert's advice. -- status: closed -> open versions: +Python 3.6, Python 3.7 ___ Python tracker

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Robert Collins added the comment: Re: backporting this. I think backporting to 3.6/3.7 makes a lot of sense - bugfix and prerelease respectively. For 2.7, this bug has been around for ages, the patch is small, and I have no objection - but the RM has already said

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Nick Coghlan added the comment: Regarding 2.7: if folks want this fixed on RHEL/CentOS, then they need to talk to Red Hat about it, not the upstream CPython devs. I used to work there, and was told multiple times by Red Hat executives that none of their customers actually

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Nir Soffer added the comment: Attaching reproducer for os.fdopen() -- Added file: https://bugs.python.org/file47492/fdopen_nfs_test.py ___ Python tracker

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Nir Soffer added the comment: Attaching reproducer for mmapobject.size() -- Added file: https://bugs.python.org/file47491/mmap_size_nfs_test.py ___ Python tracker

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Nir Soffer added the comment: Antoine, thanks for fixing this on master! but I don't think this issue can be closed yet. First, the issue is not a performance but reliability. I probably made bad choice when I marked this as performance. When you call mmap.mmap() in one

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: With Benjamin we've decided that this wouldn't happen in 2.7. Performance improvements don't warrant a backport. Thank you Nir for reporting and posting the patches! -- resolution: -> fixed stage: patch review -> resolved status:

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: New changeset d6e140466142018ddbb7541185348be2b833a2ce by Antoine Pitrou (Zackery Spytz) in branch 'master': bpo-33021: Fix GCC 7 warning (-Wmaybe-uninitialized) in mmapmodule.c (#6117)

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Zackery Spytz added the comment: Commit 4484f9dca9149da135bbae035f10a50d20d1cbbb causes GCC 7.2.0 to emit a warning. cpython/Modules/mmapmodule.c: In function ‘new_mmap_object’: cpython/Modules/mmapmodule.c:1126:18: warning: ‘fstat_result’ may be used uninitialized in this

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by Zackery Spytz : -- pull_requests: +5880 ___ Python tracker ___ ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Antoine Pitrou added the comment: New changeset 4484f9dca9149da135bbae035f10a50d20d1cbbb by Antoine Pitrou (Nir Soffer) in branch 'master': bpo-33021: Release the GIL during fstat() calls (GH-6019)

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Nir Soffer added the comment: Python cannot protect raw file descriptor from bad multi-threaded application. For example the application may close a file descriptor twice which may lead to closing unrelated file descriptor created by another thread just after it was closed,

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

pmpp added the comment: josh.r. i think you are right, i was worried if a nfs sillyrename is in progress, for eg a lock file ,then server hangs but thread lift the GIL and allow another thread to try to start to fstat the same path. --

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Josh Rosenberg added the comment: fstat is async signal safe, and I suspect it's thread safe in general, though usage does rely on the file descriptor remaining valid. If the fd in question is closed in another thread after the GIL is released, fstat would fail;

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

pmpp added the comment: is fstat thread safe ? -- nosy: +pmpp ___ Python tracker ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by Nir Soffer : -- pull_requests: +5787 ___ Python tracker ___ ___ Python-bugs-list

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

Change by Nir Soffer : -- keywords: +patch pull_requests: +5786 stage: -> patch review ___ Python tracker ___

[issue33021] Some fstat() calls do not release the GIL, possibly hanging all threads

New submission from Nir Soffer : If the file descriptor is on a non-responsive NFS server, calling fstat() can block for long time, hanging all threads. Most of the fstat() calls release the GIL around the call, but some calls seems to be forgotten. In python 3, the calls are