[Zope] Re: Zope hangs and Python eats all memory
Have you tried to enable the trace logger in the zope.conf file? Look for the stanza beginning with logger trace. You need to have your server in debug-mode. This will produce a trace.log file that can be analyzed with requestprofiler.py. This will give you the offending requests that caused Zope to hang. Now, I run the Zope server in debug now and I have enabled the trace logger. Hope to see more details. I let you know as soon as the Zope server hangs again and I get some more information. Thank you! Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope hangs and Python eats all memory
Then, this is a case where you need debugging on C level, e.g. with GDB.
You need a Python version with debug symbols (those installations
that come with the OS distribution usually lack them, you need
to build your Python yourself -- and with it your Zope).
Then you attach the Zope process (the top level process, in case
that threads show as processes as well) when the problem occurs
and analyse its threads -- especially the one that is running
when you attached.
If the problem occurs in Python (which is unlikely), then I can
provide you with GDB command definitions to analyse the Python
call chain (the Python traceback) from the C-level backtrace.
Dear Paul,
Dear Dieter,
thank you for your help.
I followed the instructions from
http://www.zopelabs.com/cookbook/1073504990.
I've built my Python myself.
After I installed GDB, I ran the following commans:
-
gdb python2.4
(gdb) info threads
(gdb) attach 6613
(gdb) thread 1
(gdb) call PyRun_SimpleString(import sys, traceback;
sys.stderr=open('/tmp/tb','w',0); traceback.print_stack())
-
Unfortunately, there is no '/tmp/tb' file written.
Below, you'll see the results after each command:
-
gdb python2.4
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as x86_64-unknown-linux-gnu...
Using host libthread_db library /lib64/libthread_db.so.1.
(gdb) attach 6613
Attaching to program: /usr/local/bin/python2.4, process 6613
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 47050976353024 (LWP 6613)]
[New Thread 1107310912 (LWP 6617)]
[New Thread 1098918208 (LWP 6616)]
[New Thread 1090525504 (LWP 6615)]
[New Thread 1082132800 (LWP 6614)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...done.
# ...a lot of Loaded / Reading here...
(gdb) info threads
5 Thread 1082132800 (LWP 6614) 0x2acae820e3fd in sem_wait ()
from /lib64/libpthread.so.0
4 Thread 1090525504 (LWP 6615) 0x2acae820e3fd in sem_wait ()
from /lib64/libpthread.so.0
3 Thread 1098918208 (LWP 6616) 0x2acae820e3fd in sem_wait ()
from /lib64/libpthread.so.0
2 Thread 1107310912 (LWP 6617) 0x2acae820e3fd in sem_wait ()
from /lib64/libpthread.so.0
1 Thread 47050976353024 (LWP 6613) 0x2acae8a3e202 in ?? () from
/lib64/libc.so.6
# (The thread 6613 is the process Id listed if I check top.
(gdb) thread 1
[Switching to thread 1 (Thread 47050976353024 (LWP 6613))]#0
0x2acae8a3e202 in ?? () from /lib64/libc.so.6
(gdb) call PyRun_SimpleString(import sys, traceback;
sys.stderr=open('/tmp/tb','w',0); traceback.print_stack())
Program received signal SIGSEGV, Segmentation fault.
0x0048fb9d in PyImport_AddModule (name=0x4d5ae9 __main__) at
Python/import.c:320
320 PyInterpreterState *interp = PyThreadState_GET()-interp;
The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use set unwindonsignal on
-
After I attached the pid 6613, Zope does not answers anymore until I
quit GDB. Is this the normal behavior?
Any idea, why no '/tmp/tb' logfile is created?
Thank you very much in advance,
Nico
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope hangs and Python eats all memory
Nico Grubert wrote: Dear list members, I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of RAM) with Python 2.4.3. installed. From time to time, Zope hangs and I cannot access it anymore. I tried to use the Zope DeadlockDebugger but if my Zope hangs I cannot call the URL http://myzopesite:8080/manage_debug_threads?secret_password; to let Zope DeadlockDebugger show any useful information. Zope does not response. If Zope hangs, the python process eats all the memory and the machine starts to swap. The top command in the shell tells me: - Tasks: 91 total, 2 running, 89 sleeping, 0 stopped, 0 zombie Cpu(s): 10.4%us, 0.4%sy, 0.0%ni, 89.0%id, 0.2%wa, 0.0%hi, 0.0%si Mem: 9041256k total, 9025124k used,16132k free,10604k buffers Swap: 4208988k total, 4208988k used,0k free, 9472k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 29992 wwwrun16 0 12.0g 8.3g 2860 S 19 95.9 3:52.66 python - There are several add-ons for Zope installed, like: - Psycopg Postgres Database Adapter - mxODBCDA ODBC Database Adapter - LDAPUserfolder - Silva Content Management System - PIL Furthermore, I see a lot of Conflict Errors in the event.log like e.g.: - ZPublisher.Conflict ConflictError at /VirtualHostBase/http/193.134.202.20:80/mysite/VirtualHostRoot/: database conflict error (oid 0x0435, class BTrees._OOBTree.OOBTree, serial this txn started with 0x036aeb14ab1c4b88 2007-01-15 12:04:40.104030, serial currently committed 0x036aeb1d0cc5d099 2007-01-15 12:13:02.993605) (80 conflicts (0 unresolved) since startup at Mon Jan 15 11:44:55 2007) - These kind of Conflict errors occur almost every minute. It might be that these errors have something to do with it but I am not sure. How can I figure out what exactly causes Zope to hang? If you need more information, please let me know. Have you tried to enable the trace logger in the zope.conf file? Look for the stanza beginning with logger trace. You need to have your server in debug-mode. This will produce a trace.log file that can be analyzed with requestprofiler.py. This will give you the offending requests that caused Zope to hang. Suresh ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope hangs and Python eats all memory
Nico Grubert wrote at 2007-1-17 10:08 +0100:
...
(gdb) thread 1
[Switching to thread 1 (Thread 47050976353024 (LWP 6613))]#0
0x2acae8a3e202 in ?? () from /lib64/libc.so.6
(gdb) call PyRun_SimpleString(import sys, traceback;
sys.stderr=open('/tmp/tb','w',0); traceback.print_stack())
Program received signal SIGSEGV, Segmentation fault.
0x0048fb9d in PyImport_AddModule (name=0x4d5ae9 __main__) at
Python/import.c:320
...
After I attached the pid 6613, Zope does not answers anymore until I
quit GDB. Is this the normal behavior?
Any idea, why no '/tmp/tb' logfile is created?
Because, the program get a SIGSEGV before it could create the file.
I found that this is often the case when you call Python functions
from GDB.
My hypothesis is that this happens because in the thread where the
Python function is called some important invariant is not met.
E.g. almost all Python functions expect to be called only when the
global interpreter lock (GIL) is held. But, this is the case
at most in a single thread of Zope's multiple threads.
For Zope debugging at C level, I therefore avoid calling
Python functions directly but use emulations on the C level.
Following are GDB commands to display the value of a Python string
object (ps) and to display the Python stack frame information
in an eval_frame (C level) call frame (pfr).
def ps
x/s ({PyStringObject}$arg0)-ob_sval
end
def pfr
ps f-f_code-co_filename
ps f-f_code-co_name
#p f-f_lineno
lineno
end
define lineno
set $__co = f-f_code
set $__lasti = f-f_lasti
set $__sz = ((PyStringObject *)$__co-co_lnotab)-ob_size/2
set $__p = (unsigned char *)((PyStringObject *)$__co-co_lnotab)-ob_sval
set $__li = $__co-co_firstlineno
set $__ad = 0
while ($__sz-1 = 0)
set $__sz = $__sz - 1
set $__ad = $__ad + *$__p
set $__p = $__p + 1
if ($__ad $__lasti)
# break -- interpreted as breakpoint
set $__sz = -1
end
if ($__sz = 0)
set $__li = $__li + *$__p
set $__p = $__p + 1
end
end
printf %d\n, $__li
end
--
Dieter
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope hangs and Python eats all memory
Nico Grubert wrote: Dear list members, I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of RAM) with Python 2.4.3. installed. From time to time, Zope hangs and I cannot access it anymore. I tried to use the Zope DeadlockDebugger but if my Zope hangs I cannot call the URL http://myzopesite:8080/manage_debug_threads?secret_password; to let Zope DeadlockDebugger show any useful information. Zope does not response. If Zope hangs, the python process eats all the memory and the machine starts to swap. The top command in the shell tells me: - Tasks: 91 total, 2 running, 89 sleeping, 0 stopped, 0 zombie Cpu(s): 10.4%us, 0.4%sy, 0.0%ni, 89.0%id, 0.2%wa, 0.0%hi, 0.0%si Mem: 9041256k total, 9025124k used,16132k free,10604k buffers Swap: 4208988k total, 4208988k used,0k free, 9472k cached Your swap appears low. Try increasing it to 20GB. Suresh ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope hangs and Python eats all memory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 16 Jan 2007, at 06:06, Suresh V wrote: Tasks: 91 total, 2 running, 89 sleeping, 0 stopped, 0 zombie Cpu(s): 10.4%us, 0.4%sy, 0.0%ni, 89.0%id, 0.2%wa, 0.0%hi, 0.0%si Mem: 9041256k total, 9025124k used,16132k free,10604k buffers Swap: 4208988k total, 4208988k used,0k free, 9472k cached Your swap appears low. Try increasing it to 20GB. How is increasing swap memory supposed to help it when one process hogs all RAM? When you start using swap heavily you are screwed already. There is no point in providing more. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFrIA9RAx5nvEhZLIRAhm3AKCCSm4m6Jon/C2/P0OmTTm3mcT9aACfTikC MlBCDml2iRTSkbo6v8wWH8Q= =WwPa -END PGP SIGNATURE- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
