[issue3618] possible deadlock in IO library (Lib/io.py)

[Benjamin Peterson]

Benjamin Peterson  added the comment:

2009/2/28 Antoine Pitrou :
>
> Antoine Pitrou  added the comment:
>
> I don't know. The RLock is a lot slower than the normal non-recursive
> variation, on the other hand I'm not sure we care about performance of
> the Python version that much. Opinions welcome.

I'm +0. The deadlock will only affect people specifically messing with
the Python io implementation. It's low priority, so maybe we should
just do it when RLock is rewritten in C.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I don't know. The RLock is a lot slower than the normal non-recursive
variation, on the other hand I'm not sure we care about performance of
the Python version that much. Opinions welcome.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Benjamin Peterson]

Benjamin Peterson  added the comment:

Yes, this is solved in the io-c branch. Antoine, do you think we should
switch _pyio to use the RLock?

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor  added the comment:

The test (io_deadlock.patch) pass on the io-c branch \o/

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Gabriel Genellina]

Changes by Gabriel Genellina :


--
nosy: +gagenellina

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor  added the comment:

I hope that this issue will be fixed by io-c (io library rewritten in 
C language).

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

(I have to add that the patch makes small reads about 60-80% slower)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Changes by Antoine Pitrou <[EMAIL PROTECTED]>:


--
keywords: +needs review

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Here is a patch. The RLock implementation is naive and as simple as
possible. It doesn't solve Haypo's case, probably because the tracing
func kicks in in the RLock code itself.

I don't want to make a decision on this alone, so someone please advise.

Added file: http://bugs.python.org/file11374/rlockio.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

We might as well bite the bullet and include a short, minimalist RLock
implementation in io.py (so as not to pull threading and all its
dependencies at startup). The C version of RLock will wait for 3.1.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Benjamin Peterson]

Changes by Benjamin Peterson <[EMAIL PROTECTED]>:


--
priority: release blocker -> critical

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Jesús Cea Avión]

Changes by Jesús Cea Avión <[EMAIL PROTECTED]>:


--
nosy: +jcea

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Selon STINNER Victor <[EMAIL PROTECTED]>:
>
> So if we consider that RLock is fast enough (see my C version of RLokc
> in #3001), we can use RLock instead of Lock to avoid this issue. Here
> is a patch to use RLock and also including an unit test of this issue.

I tried your test and it seems to lock when using the Python implementation of
RLock, but perhaps it's precisely because the implementation is in Python.

Also, your RLock implementation has to be finished before going further ;)
(see my comments in #3001)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

So if we consider that RLock is fast enough (see my C version of RLokc 
in #3001), we can use RLock instead of Lock to avoid this issue. Here 
is a patch to use RLock and also including an unit test of this issue.

Added file: http://bugs.python.org/file11190/io_deadlock.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

see the python-3000 thread I just started asking for opinions on this. 
I'd personally say this bug is a good reason to go ahead with #3001 for 3.0.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Gregory P. Smith]

Changes by Gregory P. Smith <[EMAIL PROTECTED]>:


--
nosy: +gregory.p.smith

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Barry A. Warsaw]

Changes by Barry A. Warsaw <[EMAIL PROTECTED]>:


--
priority: deferred blocker -> release blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Barry A. Warsaw]

Changes by Barry A. Warsaw <[EMAIL PROTECTED]>:


--
priority: high -> deferred blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

Ooops again, I uploaded my patch to the wrong issue! The new patch is 
now in the issue #3001.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

Changes by STINNER Victor <[EMAIL PROTECTED]>:


Removed file: http://bugs.python.org/file11174/rlock.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

Oops, I forgot to update PyInit__Thread() with my new time:
 - Add PyType_Ready()
 - Register RLockType to threading dict

Here is the new patch

--
keywords: +patch
Added file: http://bugs.python.org/file11174/rlock.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

STINNER Victor <[EMAIL PROTECTED]> added the comment:

@pitrou: I wrote an implementation of RLock in C (see #3001). So it 
would be possible to use threading.RLock instead of threading.Lock ;-)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Yes indeed. We could use an RLock to avoid the problem but RLock's are
damn slow since they are written in pure Python (see #3001). Rewriting
the critical parts of RLock (constructor, acquire(), release(),
__enter__(), __exit__()) in C should not be too complicated, would you
want to do it? :)

> Solution: use C implementation of Lib/io.py (from Python 2.6) to avoid 
> ceval hook?

I guess it's out of question. However, Buffered{Reader,Writer,Random}
should be rewritten in C one day, it is necessary for speed. Then the
problem will vanish since a lock will only need to be taken when
releasing the GIL, that is not when Python code is being interpreted.

--
nosy: +pitrou
priority:  -> high

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3618] possible deadlock in IO library (Lib/io.py)

[STINNER Victor]

New submission from STINNER Victor <[EMAIL PROTECTED]>:

BufferedWriter from Lib/io.py is thread-safe, but... the Python 
instruction "with self._write_lock:" could be interrupted when the 
lock is already acquired. Especially, _lsprof.Profiler() uses ceval 
hook and is called when the lock is acquired. But if the profiler (or 
any other module using the hook) re-use the stream (stdout or stderr), 
we get a deadlock!?

Well, this problem is really not critical since only developers 
profilers (really?).

Solution: use C implementation of Lib/io.py (from Python 2.6) to avoid 
ceval hook?

Example with py3k trunk:
 >>> import _lsprof
 >>> prof=_lsprof.Profiler(42)
 >>> prof.enable()
 # _lsprof calls 42() 
 #   -> 42 is not callable
 #   -> call PyErr_WriteUnraisable(<42 is not callable>)
 #   -> deadlock

Traceback example:

#6  0x080ecb30 in PyThread_acquire_lock (lock=0x820f550, waitflag=1) 
at Python/thread_pthread.h:352
(...)
#8  0x08162521 in PyCFunction_Call (func=0x83abfbc, arg=0xb7dc6034, 
kw=0x0) at Objects/methodobject.c:81
#9  0x080b3659 in call_function (pp_stack=0xbfbf9474, oparg=0) at 
Python/ceval.c:3403
#10 0x080ae7a6 in PyEval_EvalFrameEx (f=0x83b9f94, throwflag=0) at 
Python/ceval.c:2205
#11 0x080b3aae in fast_function (func=0x83a25b4, pp_stack=0xbfbf9724, 
n=2, na=2, nk=0) at Python/ceval.c:3491
#12 0x080b37ef in call_function (pp_stack=0xbfbf9724, oparg=1) at 
Python/ceval.c:3424
#13 0x080ae7a6 in PyEval_EvalFrameEx (f=0x83b981c, throwflag=0) at 
Python/ceval.c:2205
#14 0x080b1aee in PyEval_EvalCodeEx (co=0x838a328, globals=0x8330534, 
locals=0x0, args=0x8373da0, argcount=2, kws=0x0, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:2840
#15 0x0814ab2e in function_call (func=0x83a3b8c, arg=0x8373d8c, 
kw=0x0) at Objects/funcobject.c:628
#16 0x08118d19 in PyObject_Call (func=0x83a3b8c, arg=0x8373d8c, 
kw=0x0) at Objects/abstract.c:2181
#17 0x08132eb0 in method_call (func=0x83a3b8c, arg=0x8373d8c, kw=0x0) 
at Objects/classobject.c:323
#18 0x08118d19 in PyObject_Call (func=0x83037dc, arg=0x83141f4, 
kw=0x0) at Objects/abstract.c:2181
#19 0x080b2ed8 in PyEval_CallObjectWithKeywords (func=0x83037dc, 
arg=0x83141f4, kw=0x0) at Python/ceval.c:3283
#20 0x08141779 in PyFile_WriteObject (v=0x8398e28, f=0x83ab0dc, 
flags=1) at Objects/fileobject.c:164
#21 0x08141974 in PyFile_WriteString (s=0x819a2f2 "Exception ", 
f=0x83ab0dc) at Objects/fileobject.c:189
#22 0x080c473c in PyErr_WriteUnraisable (obj=0x81fbd78) at 
Python/errors.c:696
#23 0xb7f9612f in CallExternalTimer (pObj=0x83a7aa8) 
at /home/haypo/prog/py3k/Modules/_lsprof.c:135
#24 0xb7f96984 in Stop (pObj=0x83a7aa8, self=0x826c6d8, 
entry=0x826ec80) at /home/haypo/prog/py3k/Modules/_lsprof.c:337
#25 0xb7f96c60 in ptrace_leave_call (self=0x83a7aa8, key=0x81cb150) 
at /home/haypo/prog/py3k/Modules/_lsprof.c:420
#26 0xb7f96d5c in profiler_callback (self=0x83a7aa8, frame=0x835a0b4, 
what=6, arg=0x83ab92c) at /home/haypo/prog/py3k/Modules/_lsprof.c:471
#27 0x080b28cb in call_trace (func=0xb7f96c85 , 
obj=0x83a7aa8, frame=0x835a0b4, what=6, arg=0x83ab92c)
at Python/ceval.c:3090
#28 0x080b35da in call_function (pp_stack=0xbfbf9d74, oparg=1) at 
Python/ceval.c:3403
#29 0x080ae7a6 in PyEval_EvalFrameEx (f=0x835a0b4, throwflag=0) at 
Python/ceval.c:2205


ceval hook: Python/ceval.3403:
   PCALL(PCALL_CFUNCTION);
   if (flags & (METH_NOARGS | METH_O)) {
  ...
   } else {
  PyObject *callargs;
  callargs = load_args(pp_stack, na);
  READ_TIMESTAMP(*pintr0);
  C_TRACE(x, PyCFunction_Call(func,callargs,NULL)); <= **here**
  READ_TIMESTAMP(*pintr1);
  Py_XDECREF(callargs);
   }

--
components: Library (Lib)
messages: 71537
nosy: haypo
severity: normal
status: open
title: possible deadlock in IO library (Lib/io.py)
type: crash
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com