[issue45012] DirEntry.stat method should release GIL
uosiu added the comment: Bardzo proszę :) This fix is quite important for us. We would like to start using this fix in our product. Is there something I could do to backport it to 3.9? -- ___ Python tracker <https://bugs.python.org/issue45012> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45012] DirEntry.stat method should release GIL
Change by uosiu : -- keywords: +patch pull_requests: +26528 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28085 ___ Python tracker <https://bugs.python.org/issue45012> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45012] DirEntry.stat method should release GIL
New submission from Stanisław Skonieczny (Uosiu) : We have an application that crawls filesystem using `os.scandir`. It uses multiple threads for various things. Application is used on variety of filesystems, some of them might be slow or occasionally unresponsive. We have found out that sometimes whole crawling process is stuck and no thread makes any progress, even threads that are really simple, makes no IO and do not hold any locks. After running py-spy on process that was stuck we saw that one of the threads has entered `dentry.stat(follow_symlinks=False)` line and still holds the GIL. Other threads are stuck, because they are waiting for the GIL. This situation can take a long time. I think that `DirEntry` should release GIL when stat cache is empty and syscall is performed. This bug has already been fixed in `scandir` module. See: https://github.com/benhoyt/scandir/issues/131 -- components: Library (Lib) messages: 400337 nosy: Stanisław Skonieczny (Uosiu) priority: normal severity: normal status: open title: DirEntry.stat method should release GIL type: behavior versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue45012> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26600] MagickMock __str__ sometimes returns MagickMock instead of str
Stanisław Skonieczny (Uosiu) added the comment: When you just create a magick mock and then call `__str__` on it, it never throws an exception and always return a string. But... when you create a magick mock and then call `__str__` from two threads concurrently, one of them will raise an exception. Race is as follows: 1. Thread 1 creates magick mock m. 2. Thread 1 enters `m.__str__` it starts some internal preparations. It does not exit `__str__` yet. 3. Thread 2 enters `m.__str__`. In effect one of this threads will try to return `MagickMock` instance instead of `str` object, which if forbidden by python. Some low-level python check will raise following exception: `TypeError: __str__ returned non-string (type MagicMock)`. Stanisław Skonieczny On Mon, Feb 22, 2021 at 10:56 AM Irit Katriel wrote: > > Irit Katriel added the comment: > > Why is this only relevant to multiple thread? > > -- > > ___ > Python tracker > <https://bugs.python.org/issue26600> > ___ > -- ___ Python tracker <https://bugs.python.org/issue26600> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26600] MagickMock __str__ sometimes returns MagickMock instead of str
Stanisław Skonieczny (Uosiu) added the comment: However there is still nothing in the docs that you have to mock `__str__` by yourself when using mock object with multiple threads. Otherwise calling `__str__` may result with raising exception if you are unlucky. Stanisław Skonieczny On Wed, Feb 17, 2021 at 7:26 PM Irit Katriel wrote: > > Irit Katriel added the comment: > > The documentation has examples how to set up a mock's __str__. > > https://docs.python.org/3/library/unittest.mock.html > > -- > nosy: +iritkatriel > resolution: -> not a bug > stage: -> resolved > status: open -> closed > > ___ > Python tracker > <https://bugs.python.org/issue26600> > ___ > -- ___ Python tracker <https://bugs.python.org/issue26600> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34486] "RuntimeError: release unlocked lock" when starting a thread
Stanisław Skonieczny (Uosiu) added the comment: This scripts runs start_threads.py and send a signal to it. After some number of loops problem can be reproduced. -- Added file: https://bugs.python.org/file47762/test.sh ___ Python tracker <https://bugs.python.org/issue34486> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34486] "RuntimeError: release unlocked lock" when starting a thread
New submission from Stanisław Skonieczny (Uosiu)
:
Sometimes when thread is starting it raises "RuntimeError: release unlocked
lock". That is when signal handler is invoked in the same time.
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 551, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.6/threading.py", line 304, in wait
self._acquire_restore(saved_state)
File "start_threads.py", line 12, in sighandler
raise MyException("got signal")
__main__.MyException: got signal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "start_threads.py", line 28, in
thread.start()
File "/usr/lib/python3.6/threading.py", line 851, in start
self._started.wait()
File "/usr/lib/python3.6/threading.py", line 552, in wait
return signaled
File "/usr/lib/python3.6/threading.py", line 243, in __exit__
return self._lock.__exit__(*args)
RuntimeError: release unlocked lock
I have attached to files that reproduce the problem. First runs python and
starts noop threads in the loop. It uses signal handler that raises exception.
Second is a bash script to run python script and send signal to it. After some
time, when signal is handled in unfortunate place, python script breaks due to
unreleased lock.
--
components: Library (Lib)
files: start_threads.py
messages: 323998
nosy: Stanisław Skonieczny (Uosiu)
priority: normal
severity: normal
status: open
title: "RuntimeError: release unlocked lock" when starting a thread
versions: Python 3.6
Added file: https://bugs.python.org/file47761/start_threads.py
___
Python tracker
<https://bugs.python.org/issue34486>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26600] MagickMock __str__ sometimes returns MagickMock instead of str
New submission from Stanisław Skonieczny (Uosiu): This bug results in raising TypeError: __str__ returned non-string (type MagicMock) Following program can reproduce it: ``` from threading import Thread from mock.mock import MagicMock def main(): mocks = [MagicMock() for _ in range(1000)] def in_thread(): for m in mocks: str(m) threads = [Thread(target=in_thread) for _ in range(10)] for thread in threads: thread.start() for thread in threads: thread.join() if __name__ == '__main__': main() ``` For me it rasies few times: ``` Exception in thread Thread-6: Traceback (most recent call last): File "/usr/local/lib/python3.5/threading.py", line 923, in _bootstrap_inner self.run() File "/usr/local/lib/python3.5/threading.py", line 871, in run self._target(*self._args, **self._kwargs) File "../misc/magic_mock_str_bug.py", line 11, in in_thread str(m) TypeError: __str__ returned non-string (type MagicMock) ``` -- components: Library (Lib) messages: 262104 nosy: Stanisław Skonieczny (Uosiu) priority: normal severity: normal status: open title: MagickMock __str__ sometimes returns MagickMock instead of str type: behavior versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26600> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25731] Assigning and deleting __new__ attr on the class does not allow to create instances of this class
New submission from Stanisław Skonieczny (Uosiu):
When moving from python 2.7 to 3.5 I have found a problem with patching __new__
method on the class. It was done this way:
'''
patch('foo.bar.MyClass.__new__', return_value=mocked_instance)
'''
In works with python 2.7, but in 3.5 it fails with:
'''
TypeError: object() takes no parameters
'''
I have created minimal scenario that ilustrates this bug cause:
'''
class X:
def __init__(self, a):
pass
def new(cls, a):
pass
X(1)
X.__new__ = new
X(1)
del X.__new__
X(1)
'''
Setting __new__ attribute and then deleting it has some side effect.
--
components: Interpreter Core
files: new_patch_fails.py
messages: 255337
nosy: Stanisław Skonieczny (Uosiu)
priority: normal
severity: normal
status: open
title: Assigning and deleting __new__ attr on the class does not allow to
create instances of this class
type: behavior
versions: Python 3.5
Added file: http://bugs.python.org/file41161/new_patch_fails.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25731>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
