[issue9670] Exceed Recursion Limit in Thread

[Jared Lang]

New submission from Jared Lang :

Recursion within a thread on OSX can result in a crash by exceeding the systems 
recursion limit.  Recursion behaves as expected if not in thread, meaning it 
throws a RunTimeError with the message "maximum recursion depth exceeded."

The crash is able to be avoided if the recursion limit is set to a lower 
number, ie. 800, via sys.setrecursionlimit.

Example program which crashes on OSX:


>>> def f():
... return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> thread.start()
Bus error


Alternatively, the following works as expected:


>>> import sys
>>> def f():
... return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> sys.setrecursionlimit(800)
>>> thread.start()

>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
 line 532, in __bootstrap_inner
self.run()
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
 line 484, in run
self.__target(*self.__args, **self.__kwargs)
  File "", line 2, in f
  File "", line 2, in f
  File "", line 2, in f
  File "", line 2, in f
  File "", line 2, in f
  File "", line 2, in f
...

RuntimeError: maximum recursion depth exceeded

--
assignee: ronaldoussoren
components: Macintosh
messages: 114787
nosy: jaredlang, ronaldoussoren
priority: normal
severity: normal
status: open
title: Exceed Recursion Limit in Thread
type: crash
versions: Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[R. David Murray]

R. David Murray  added the comment:

This is not the first recursion limit related problem we've seen with FreeBSD 
derived systems.  See for example Issue1201456. It would be nice to have an 
actual fix for this class of problem, but I have no clue what that would look 
like.  There may also be a specific fix for this issue.  At least, we can hope 
:)

Python 2.6 is in security-fix-only mode, so I've updated the versions.  I can 
confirm that this reproduces on 2.7 and py3k trunk on OS/X 10.4 using a 
non-framework build.  I haven't tested 3.1 but I am sure it must behave the 
same.

--
nosy: +r.david.murray
stage:  -> needs patch
versions: +Python 2.7, Python 3.1, Python 3.2 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

This is the same issue as #6763.

The root cause of this issue is that the default stack size for threads other 
than the main thread is small. 

One way to fix this issue is to increase the default stacksize for new threads.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[R. David Murray]

R. David Murray  added the comment:

As I mentioned on the other ticket, I think bumping the default stack size is 
probably the most useful solution.  This is is a clearer description of the 
stack problem, since it doesn't get involved in MIMETypes internals, so I'm not 
closing it as a duplicate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The attached patch explicitly sets the minimal stack size to about 704K, which 
is the minimal size where 'def f(): f()' doesn't cause a buserror when run in a 
thread.

--
keywords: +needs review, patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file19298/issue9670.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Leonardo Santagada]

Leonardo Santagada  added the comment:

Why not make it bigger so it doesn't crash for much bigger functions?

--
nosy: +santagada

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

On 20 Oct, 2010, at 17:41, Leonardo Santagada wrote:

> 
> Leonardo Santagada  added the comment:
> 
> Why not make it bigger so it doesn't crash for much bigger functions?

I don't mind making it bigger, but a larger size does come with a cost: the 
larger the initial stack size the less threads you can have without trashing 
the system (or running out of address space on a 32-bit build). 

I'm fine with increasing it to 1M, which is a nice round number build not 
larger. 

BTW. I haven't looked up the default stack size for the main thread on OSX.

Ronald
> 
> --
> nosy: +santagada
> 
> ___
> Python tracker 
> 
> ___

--
Added file: http://bugs.python.org/file19301/smime.p7s

___
Python tracker 

___

smime.p7s
Description: S/MIME cryptographic signature
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Leonardo Santagada]

Leonardo Santagada  added the comment:

The default stack size in osx is 8MB. I think it is a safer bet, and no one is 
supposed to be using more than like 40-50 threads anyway (specially in 32 bit 
only systems limited to 8 cores).

Is there a user case for hundreds/thousands of threads in python?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[R. David Murray]

R. David Murray  added the comment:

It would be nice if we could expand this fix to include FreeBSD, which as I 
understand it has the same problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Here's an updated combined patch.  I've revised Ronald's test to run via 
subprocess so it should be safe to add to the standard tests.  It works as 
expected on OS X.  If there are no objections, I will commit it and monitor the 
buildbots.

--
nosy: +ned.deily
stage: patch review -> commit review
Added file: http://bugs.python.org/file21527/issue9670-v3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Running the test in a separate process is a good idea.

The patch should be fine and fixes a problem on OSX, if the buildbot for 
FreeBSD starts to complain we can always removing the #if that tests for that 
platform.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Roundup Robot]

Roundup Robot  added the comment:

New changeset b0d2b696da19 by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/b0d2b696da19

New changeset 378b40d71175 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/378b40d71175

New changeset 3fe8fd2fd1d0 by Ned Deily in branch '3.2':
Issue #9670: merge with 3.2
http://hg.python.org/cpython/rev/3fe8fd2fd1d0

New changeset 4c750091d8c5 by Ned Deily in branch 'default':
Issue #9670: merge with current
http://hg.python.org/cpython/rev/4c750091d8c5

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Applied in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and 
default (for 3.3).

--
resolution:  -> fixed
stage: commit review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Looks like the patch breaks the OpenIndiana buildbots:

http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.2/builds/168

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 42d5001e5845 by Ned Deily in branch '3.1':
Issue9670: Back out changeset 378b40d71175; test fails on other platforms
http://hg.python.org/cpython/rev/42d5001e5845

New changeset 54edabf2846d by Ned Deily in branch '3.2':
Issue9670: Merge backout to 3.2.
http://hg.python.org/cpython/rev/54edabf2846d

New changeset b7456c1b4aa4 by Ned Deily in branch 'default':
Issue9670: Merge backout from 3.2.
http://hg.python.org/cpython/rev/b7456c1b4aa4

New changeset 3630bc3d5a88 by Ned Deily in branch '2.7':
Issue9670: Back out changeset b0d2b696da19; test fails on other platforms
http://hg.python.org/cpython/rev/3630bc3d5a88

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Reverting the patch since it caused failures on failure on some other platform 
buildbots (for instance, Gentoo and OpenIndiana).  It also fails on OS X 
buildbots with pydebug enabled, something I hadn't tested:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.2/builds/161

So the patch needs to address the stack size for OS X pydebug builds and what 
to do about other failing platforms - increase the stack size for each or 
disable the test there.

--
stage: committed/rejected -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

For pydebug builds the default stacksize should be increased, and prefer to do 
this by unconditionally changing the stacksize.

W.r.t. the non-osx buildbot failures: it would be better if we could come up 
with a solution that doesn't disable the test for those platform but I won't do 
that work. I therefore propose to only enable the test on OSX.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Version 4 of the patch has two changes w.r.t version 3:

* Increase the stack size on OSX to ensure that tests pass with
  --with-pydebug

* Only test the recursion behavior on OSX (where we know the bug is
  fixed)

--
Added file: http://bugs.python.org/file21907/issue9670-v4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Roundup Robot]

Roundup Robot  added the comment:

New changeset ecf0ef85c72a by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/ecf0ef85c72a

New changeset 0cded2f2cea3 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/0cded2f2cea3

New changeset 8a484090da7e by Ned Deily in branch '3.2':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8a484090da7e

New changeset 8220f68f1229 by Ned Deily in branch 'default':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8220f68f1229

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Version 4 looks good and the tests pass on OS X with pydebug enabled.  Applied 
in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and default 
(for 3.3).

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Changes by Ronald Oussoren :


Removed file: http://bugs.python.org/file19301/smime.p7s

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've slightly updated the patch:

1) Increase default stack size to 1MByte

2) Change the preprocessor guard to also trigger on FreeBSD

Without this patch I get crashes for 32-bit and 64-bit Intel architectures on 
OSX 10.6, with this patch those crashes go away.

I don't have access to a freebsd box so cannot test there.

--
Added file: http://bugs.python.org/file21126/issue9670-v2.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The attached unittest file triggers the issue until the patch is applied.

I'm not sure if I'd be a good idea to add this test as is to the testsuite, 
given the hard crash when it fails.

--
Added file: http://bugs.python.org/file21127/test_issue9670.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Re-opening. The test fails on 3.2 on the new Lion buildbot (but neither on 3.3 
nor 2.7, it seems).
See http://buildbot.python.org/all/builders/AMD64%20Lion%203.2

--
nosy: +lukasz.langa, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

3x is now using clang to compile on 10.7 Lion; 32 is failing back to the Xcode 
4 default of llvm-gcc which has proved problematic fox 3x.  Rather than tweak 
the test again, try appending a CC=clang to the ./configure for the 3.2 
buildbot until the clang changes are backported.  The builds should run faster, 
too, with clang.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> try appending a CC=clang to the ./configure for the 3.2 buildbot until
> the clang changes are backported.

Ok, done that. Can someone watch the next builds?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9670] Exceed Recursion Limit in Thread

[Ned Deily]

Ned Deily  added the comment:

Looks like using clang brought the size back down again.  Closing.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com