[issue45012] DirEntry.stat method should release GIL

[Łukasz Langa]

Łukasz Langa  added the comment:

Sadly, we can't backport this to 3.9 as it's only accepting bugfixes and this 
is a performance improvement. 3.10 is out of scope for this too as 3.10.0rc2 
shipped last night.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45012] DirEntry.stat method should release GIL

[uosiu]

uosiu  added the comment:

Bardzo proszę :)

This fix is quite important for us. We would like to start using this fix in 
our product. Is there something I could do to backport it to 3.9?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45012] DirEntry.stat method should release GIL

[Łukasz Langa]

Łukasz Langa  added the comment:

Dzięki za zgłoszenie i poprawkę, Stanisław! ✨  ✨

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
type: behavior -> performance
versions:  -Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45012] DirEntry.stat method should release GIL

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 9dc363ee7cf2eb6ff374fbf7bbeb0b333f4afb8f by Stanisław Skonieczny 
in branch 'main':
bpo-45012: Release GIL around stat in os.scandir (GH-28085)
https://github.com/python/cpython/commit/9dc363ee7cf2eb6ff374fbf7bbeb0b333f4afb8f


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45012] DirEntry.stat method should release GIL

[uosiu]

Change by uosiu :


--
keywords: +patch
pull_requests: +26528
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28085

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45012] DirEntry.stat method should release GIL

[Uosiu]

New submission from Stanisław Skonieczny (Uosiu) 
:

We have an application that crawls filesystem using `os.scandir`. It uses 
multiple threads for various things. Application is used on variety of 
filesystems, some of them might be slow or occasionally unresponsive.

We have found out that sometimes whole crawling process is stuck and no thread 
makes any progress, even threads that are really simple, makes no IO and do not 
hold any locks. After running py-spy on process that was stuck we saw that one 
of the threads has entered `dentry.stat(follow_symlinks=False)` line and still 
holds the GIL. Other threads are stuck, because they are waiting for the GIL. 
This situation can take a long time.

I think that `DirEntry` should release GIL when stat cache is empty and syscall 
is performed.

This bug has already been fixed in `scandir` module. See: 
https://github.com/benhoyt/scandir/issues/131

--
components: Library (Lib)
messages: 400337
nosy: Stanisław Skonieczny (Uosiu)
priority: normal
severity: normal
status: open
title: DirEntry.stat method should release GIL
type: behavior
versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 
3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com