[issue7205] bz2file deadlock
James G. sack (jim) jgs...@users.sourceforge.net added the comment: On my Fedora 11 AMD x86_64 system, it appears the deadlock still occurs (up to the limit of my patience, ie: several minutes). If I reduce to say 3, I can get the test to succeed sometimes. Observed in: trunk -r 76850 (latest) -r 75818 (first occurrence of testThreading in test-bz2.py) but not -r 75817 ~jim -- nosy: +jgsack ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
James G. sack (jim) jgs...@users.sourceforge.net added the comment: IMPORTANT Correction: Please disregard msg96472. I was forgetting to do .configure and make, and evidently getting bogus failures. test_bz2 works fine now, ..sorry for the false alarm. ~jim -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Antoine Pitrou pit...@free.fr added the comment: Fixed in r75818 (trunk), r75819 (2.6), r75820 (py3k), r75821 (3.1). Thanks! -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Antoine Pitrou pit...@free.fr added the comment: Here is a patch. -- keywords: +patch Added file: http://bugs.python.org/file15205/bzthreads.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Robert Collins robe...@robertcollins.net added the comment: On Mon, 2009-10-26 at 19:23 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: Here is a patch. Looks fine to me assuming that the locking functions can be used outside the GIL. On the test side, the case I supplied was low noise for me - I'd hesitate to do as much compression as you are (50 times more) unless you saw it spuriously pass a lot - the nature of the deadlock isn't dependent on races so much as simple scheduling - as long as the seocnd thread is scheduled before the first thread completes compressing the deadlock will occur. -Rob -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Antoine Pitrou pit...@free.fr added the comment: Looks fine to me assuming that the locking functions can be used outside the GIL. Yes, they can. Actually, even the GIL uses them. :-) On the test side, the case I supplied was low noise for me - I'd hesitate to do as much compression as you are (50 times more) unless you saw it spuriously pass a lot - the nature of the deadlock isn't dependent on races so much as simple scheduling - as long as the seocnd thread is scheduled before the first thread completes compressing the deadlock will occur. Well, your test case often succeeded here, so I decided on a more aggressive variation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Robert Collins robe...@robertcollins.net added the comment: On Mon, 2009-10-26 at 21:27 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: Well, your test case often succeeded here, so I decided on a more aggressive variation. fair enough, if its needed - its needed :) -Rob -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
New submission from Robert Collins robe...@robertcollins.net: There is a systemic bug in BZ2File where the GIL is released to perform compression work, and any other thread calling into BZ2File will deadlock. We noticed in the write method, but inspection of the code makes it clear that its systemic. The problem is pretty simple. Say you have two threads and one bz2file object. One calls write(), the other calls (say) seek(), but it could be write() or other methods too. Now, its pretty clear that the question 'should these two threads get serialised' could be contentious. So lets put that aside by saying 'raising an exception or serialising in arbitrary order would be ok'. What happens today is: t1:bz2file.write bz2file.lock.acquire gil-release bz2compression starts t2:gil-acquired bz2file.seek bz2file.lock.acquire(wait=1) - this thread is stuck now, and has the GIL t1:bz2compression finishes gil.acquire - this thread is stuck now, waiting for the GIL If any owner of the bz2file object lock will release the GIL, *all* routines that attempt to lock the bz2file object have to release the GIL if they can't get the lock - blocking won't work. I'm not familiar enough with the python threading API to know whether its safe to call without the GIL. If its not then clearly it can't be used to work with getting the GIL, and lower layer locks should be used. -- components: Extension Modules files: bz2fail.py messages: 94462 nosy: barry, rbcollins, statik severity: normal status: open title: bz2file deadlock type: crash Added file: http://bugs.python.org/file15196/bz2fail.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Antoine Pitrou pit...@free.fr added the comment: Thanks, nice catch. -- nosy: +pitrou priority: - high stage: - needs patch versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Robert Collins robe...@robertcollins.net added the comment: On Sun, 2009-10-25 at 22:00 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: Thanks, nice catch. Yeah :). versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 Python 2.5 is also affected - its what we're running on the server that broke :) -Rob -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Antoine Pitrou pit...@free.fr added the comment: Python 2.5 is also affected - its what we're running on the server that broke :) Yes, but it doesn't receive any bug fixes anymore -- only security fixes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7205] bz2file deadlock
Robert Collins robe...@robertcollins.net added the comment: On Sun, 2009-10-25 at 22:27 +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: Python 2.5 is also affected - its what we're running on the server that broke :) Yes, but it doesn't receive any bug fixes anymore -- only security fixes. Ok, we'll work around the issue there then ;) -Rob -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
