Re: Segmentation faults using threads
Mathias wrote:
>>
>> What module are you using for SSH?
>>
>> What's in your program that isn't pure Python?
>> The problem is probably in some non-Python component; you shouldn't
>> be able to force a memory protection error from within Python code.
>>
>
> It looks like the error could be in scipy/Numeric, when a large array's
> type is changed, like this:
>
> >>> from scipy import *
> >>> a=zeros(1,'b')#100 MiB
> >>> b=a.copy().astype('d') #800 MiB, ok
> >>> a=zeros(10,'b')#1GiB
> >>> b=a.copy().astype('d') #8GiB, fails with sf
> Segmentation fault
>
> if I use zeros directly for allocation of the doubles it works as expected:
>
> >>> from scipy import *
> >>> a=zeros(10,'d')#8GiB, fails with python exception
> Traceback (most recent call last):
> File "", line 1, in ?
> MemoryError: can't allocate memory for array
> >>>
>
> I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
> scipy version 0.3.2, Numeric v 24.2
That sounds like the case where the array has to be reallocated from
4-byte floats to 8-byte doubles is being botched.
Take a look at
"http://www.mail-archive.com/numpy-discussion@lists.sourceforge.net/msg02033.html";
and then at array_cast in arrraymethods.c of scipy. There may be a
reference count bug in that C code. I'm not familiar enough with
Python reference count internals to be sure, though.
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
>
> What module are you using for SSH?
>
> What's in your program that isn't pure Python?
> The problem is probably in some non-Python component; you shouldn't
> be able to force a memory protection error from within Python code.
>
It looks like the error could be in scipy/Numeric, when a large array's
type is changed, like this:
>>> from scipy import *
>>> a=zeros(1,'b') #100 MiB
>>> b=a.copy().astype('d') #800 MiB, ok
>>> a=zeros(10,'b')#1GiB
>>> b=a.copy().astype('d') #8GiB, fails with sf
Segmentation fault
if I use zeros directly for allocation of the doubles it works as expected:
>>> from scipy import *
>>> a=zeros(10,'d')#8GiB, fails with python exception
Traceback (most recent call last):
File "", line 1, in ?
MemoryError: can't allocate memory for array
>>>
I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
scipy version 0.3.2, Numeric v 24.2
--
http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
"Mathias" <[EMAIL PROTECTED]> wrote: > Does someone have experience with threading in python - are there > non-threadsafe functions I should know about? how do your threads communicate with one another - are there any globals that are accessed from different threads? strange this - you should get an exception, not a segment fault if its in the python bits.. - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
PS: setting sys.setcheckinterval(1) reduces the probablilty of a failure as well, but definetely at a performance cost. -- http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
John Nagle wrote: > Daniel Nogradi wrote: >>> I use the thread module (not threading) for a client/server app where I >>> distribute large amounts of pickled data over ssh tunnels. > > What module are you using for SSH? > > What's in your program that isn't pure Python? > The problem is probably in some non-Python component; you shouldn't > be able to force a memory protection error from within Python code. > > Also note that the "marshal" module may be unsafe. > > John Nagle I'm using os.popen2() to pipe into an ssh session via stdin/stdout. That's probably not the elegant way... Other modules: scipy 0.3.2 (with Numeric 24.2) and python 2.4 Does pickle/cPickle count as part of the marshal module? Mathias -- http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
Daniel Nogradi wrote: >> I use the thread module (not threading) for a client/server app where I >> distribute large amounts of pickled data over ssh tunnels. What module are you using for SSH? What's in your program that isn't pure Python? The problem is probably in some non-Python component; you shouldn't be able to force a memory protection error from within Python code. Also note that the "marshal" module may be unsafe. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
> Hi, it would be helpful if you posted a minimalistic code snippet > which showed the problem you describe. > > Daniel I wish I could! If I knew exactly where the effect takes place I could probably circumvent it. All I know know is that it happens under high load and with a lot of waitstates I can reduce the propability of crashing. So there must be some race condition somewhere I think. Is there a way to analyze where the crash took place? I guess I can have a core dumped and somehow analyze it, but that's probably very hard to do. Would a profiler work which records the function call structure? Does someone have experience with threading in python - are there non-threadsafe functions I should know about? Thanks, Mathias -- http://mail.python.org/mailman/listinfo/python-list
Re: Segmentation faults using threads
> I use the thread module (not threading) for a client/server app where I > distribute large amounts of pickled data over ssh tunnels. > Now I get regular Segmentation Faults during high load episodes. I use a > semaphore to have pickle/unpickle run nonthreaded, but I still get > frequent nondeterministic segmentation faults. > Since there is no traceback after a sf, I have no clue what exactly > happened, and debugging a multithreaded app is no fun anyway :( > > Can someone recommend me how to get extra info during such a crash? > Or any other opinion on where the problem might lie? Hi, it would be helpful if you posted a minimalistic code snippet which showed the problem you describe. Daniel -- http://mail.python.org/mailman/listinfo/python-list
