Re: ulimit stack size and python threads
> I see. I should be blaming the default behavior of pthreads. You shouldn't blame anybody. Instead, you should sit down and study the problem in detail, until you fully understand it. Then you should start contributing fixes. Never ever should you spread blame. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> Always crashing because I asked the OS to please not allow a process > to grow too big is what I call overloading the meaning of ulimit -s. Please trust that there is no explicit code in the Python interpreter that tests whether the stack size is 4GB, and then produces an explicit crash. > It's quite surprising. Not to mention the poor error message. AFAICT, you didn't even *report* yet what the error message is, so I don't know whether it is poor. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
I see. I should be blaming the default behavior of pthreads. I did work on a OpenMP library once, and we worked around this problem, plus we gave good error messages. Given the number of HPC sites which use Python, I'd think that Python would have grown similar features. (HPC sites are more likely to have intermediate-sized stack limits due to use of Fortran.) -- greg -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
Greg Lindahl wrote: I figure this is a FAQ, but I can't find it in any FAQs. I want to limit the stacksize on my server. If I set it to 8 megs, or unlimited, python is happy. If I set it to 4 gigabytes, things like yum (which is a python program) crash creating a thread. This is on an x86_64 linux kernel, RHEL5, etc etc. Why is Python overloading the meaning of the ulimit -s like this? There are plenty of real non-python programs with huge stack usage, and I'd like my system default stack limit to be less than unlimited but much larger than Python will allow. The Python interpreter itself (absent a call to resource.setrlimit()) does nothing with resource limits - it just uses the environment it is loaded into. In the absence of effective alternative solutions, it may be possible to achieve the effect you desire by calling resource.setrlimit() in your Python installation's site.py, effectively over-riding the system default. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> Why do you think Python is overloading the meaning of that? I ensure > you it isn't - it doesn't actively care what the limits are. Always crashing because I asked the OS to please not allow a process to grow too big is what I call overloading the meaning of ulimit -s. It's quite surprising. Not to mention the poor error message. -- greg -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> But even if that worked, I'd be worried that python is doing something > bad with the ulimit -s value under the covers. Again: it definitely isn't. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> How much higher? You could try just under 4GB (unsigned 32-bit) and just > under 2GB (signed 32-bit). I'd like to set it to be about 1/2 the memory size of my server, which happens to end up being 4 gbytes. And no, slightly less than 4 gb doesn't work. But even if that worked, I'd be worried that python is doing something bad with the ulimit -s value under the covers. -- greg -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> Why is Python overloading the meaning of the ulimit -s like this? Why do you think Python is overloading the meaning of that? I ensure you it isn't - it doesn't actively care what the limits are. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
Greg Lindahl wrote: I'm only guessing, but could it be a 32-bit limit somewhere? Have you tried, say, 1GB, which would be within a 32-bit limit? Indeed, ulimit -s 100 (a bit smaller than 1 GB) does work, but it doesn't solve my problem, since I want to set the limit higher than 1 GB. How much higher? You could try just under 4GB (unsigned 32-bit) and just under 2GB (signed 32-bit). -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
> I'm only guessing, but could it be a 32-bit limit somewhere? Have you > tried, say, 1GB, which would be within a 32-bit limit? Indeed, ulimit -s 100 (a bit smaller than 1 GB) does work, but it doesn't solve my problem, since I want to set the limit higher than 1 GB. -- greg -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
Greg Lindahl wrote: I figure this is a FAQ, but I can't find it in any FAQs. I want to limit the stacksize on my server. If I set it to 8 megs, or unlimited, python is happy. If I set it to 4 gigabytes, things like yum (which is a python program) crash creating a thread. This is on an x86_64 linux kernel, RHEL5, etc etc. Why is Python overloading the meaning of the ulimit -s like this? There are plenty of real non-python programs with huge stack usage, and I'd like my system default stack limit to be less than unlimited but much larger than Python will allow. I'm only guessing, but could it be a 32-bit limit somewhere? Have you tried, say, 1GB, which would be within a 32-bit limit? -- http://mail.python.org/mailman/listinfo/python-list
ulimit stack size and python threads
I figure this is a FAQ, but I can't find it in any FAQs. I want to limit the stacksize on my server. If I set it to 8 megs, or unlimited, python is happy. If I set it to 4 gigabytes, things like yum (which is a python program) crash creating a thread. This is on an x86_64 linux kernel, RHEL5, etc etc. Why is Python overloading the meaning of the ulimit -s like this? There are plenty of real non-python programs with huge stack usage, and I'd like my system default stack limit to be less than unlimited but much larger than Python will allow. -- greg -- http://mail.python.org/mailman/listinfo/python-list
