[issue14432] Bug in generator if the generator in created in a C thread

[STINNER Victor]

STINNER Victor added the comment:

Updated patch for Python 3.4:

- remove PyFrameObject.f_tstate attribute: the thread state can be easily 
retrieved, it is known where it is needed (see the patch). There is one 
function which doesn't know the thread state: _PyEval_CallTracing(), but this 
function was already calling PyEval_GetFrame() which calls PyThreadState_GET() 
internally, so...

- add an unit test for this issue (generator created in a temporary C thread)

It's really hard to reproduce the crash. I tried with my old tarball and I 
failed. I also tried with my unit test and I failed. I'm pretty sure that the 
crash can only be reproduced when Python is compiled is release mode.

I reproduced the crash once with the unit test on an unpatched Python 3.4.


For Python 2.7 and 3.3, what do you think of applying generator.patch? It looks 
simple and obvious. I don't know the impact on performances, but it should be 
very low.

--
nosy: +serhiy.storchaka
Added file: http://bugs.python.org/file32605/issue14432.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[STINNER Victor]

STINNER Victor added the comment:

ping? (for myself :-))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Andrew Suffield]

Andrew Suffield  added the comment:

I think I've tripped over a variation on this theme using pyqt and 2.7:

When working in a QThread, the PyGILState_Ensure call when transitioning 
control from Qt to python will frequently allocate a new thread state - because 
every time control returns to the Qt event loop, gilstate_counter is likely to 
become zero.

If a generator is kept around longer than this, it becomes quite likely to have 
an older thread state in f_tstate. This happens all the time.

The access that blows up on me is PyEval_GetRestricted, since 
PyFrame_IsRestricted checks f_tstate. Annoyingly this debris is still called on 
the possibly-restricted operations even when restricted execution is nowhere in 
sight.

Hence, any use of open() in a long-lived generator in a QThread is probably 
going to crash.

Patch against 2.7?

--
nosy: +asuffield

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Mark Shannon]

Mark Shannon  added the comment:

Rather than ensuring that f_tstate always points to the current frame,
just remove it altogether.

Patch attached

--
nosy: +Mark.Shannon
Added file: http://bugs.python.org/file25176/remove_f_tstate.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Adam Tomjack]

Adam Tomjack  added the comment:

For what it's worth, I think I've seen this bug in 2.6 and 2.5, using 
generators created in python threads, while profiling not tracing.

I'm creating generators in one python thread and storing them in a variable.  
In a different python thread I overwrite the variable, which garbage collects 
the generator.  Sometimes it causes an interpreter crash.  Other times, it 
seems to trigger a return event in the profiler, but it's associated with an 
incorrect thread.  The thread in question is often (always?) in the profiler 
code, so it looks like the profiler is profiling itself.

--
nosy: +adamtj
versions: +Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Andrew Svetlov]

Changes by Andrew Svetlov :


--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Nick Coghlan]

Nick Coghlan  added the comment:

It may not even have to specifically test the crash - any operation that 
accessed the tstate on the frame and could be shown to be accessing the wrong 
thread state when called from another thread could demonstrate the problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Nick Coghlan]

Nick Coghlan  added the comment:

The proposed fix sounds reasonable to me. Would it be possible to work 
something into test_capi to actually test it?

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[Ross Lagerwall]

Ross Lagerwall  added the comment:

Here's the patch ;-)

--
keywords: +patch
nosy: +rosslagerwall
Added file: http://bugs.python.org/file25055/generator.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14432] Bug in generator if the generator in created in a C thread

[STINNER Victor]

New submission from STINNER Victor :

We have a crash in our product when tracing is enabled by
sys.settrace() and threading.settrace(). If a Python generator is
created in a C thread, calling the generator later in another thread
may crash if Python tracing is enabled.

 - the C thread calls PyGILState_Ensure() which creates a temporary
Python thread state
 - a generator is created, the generator has a reference to a Python
frame which keeps a reference to the temporary Python thread state
 - the C thread calls PyGILState_Releases() which destroys the
temporary Python thread state
 - when the generator is called later in another thread, call_trace()
reads the Python thread state from the generator frame, which is the
destroyed frame => it does crash on a pointer dereference if the
memory was reallocated (by malloc()) and the data were erased

To reproduce the crash, unpack the attached
generator_frame_bug.tar.gz, compile the C module using:

   python setup.py build

and then run:

   PYTHONPATH=$(ls -d build/lib*/) python test.py

(or just "python test.py if you installed the _test module).
You may need to use Valgrind to see the error, or call memset(tstate,
0xFF, sizeof(*tstate)) before free(tstate) in tstate_delete_common().

Calling the generator should update its reference to the Python state
thread in its frame. The generator may also clears frame->f_tstate (to
detect bugs earlier), as it does for frame->f_back (to avoid a
reference cycle). Attached patch implements this fix for Python 3.3.

--
components: Interpreter Core
files: generator_frame_bug.tar.gz
messages: 156982
nosy: haypo, neologix, pitrou
priority: normal
severity: normal
status: open
title: Bug in generator if the generator in created in a C  thread
type: crash
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file25054/generator_frame_bug.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com