Hi
I was bitten by the same problem and went so far to track down all the
fishy cycles which easily can be created. Most of the cycles I detected
were multi-object cycles which were created by me just storing
references to objects to have them handy. Certain types of cycles cannot
be broken by the gc, you get a runaway situation either in Python24 or
Python25. My code now runs without problems for months serving more than
10k users a day providing a card gaming environment.
Things to check:
- the object hierarchy on the server and the client should be in sync,
if not detach might not work correctly
- do you use stateful LivePages?
if so, check the toremember list in the request if its growing ad libidum
- do you use guard/sessions?
if so, be careful with the session/mind object and what you store in mind
But without seeing your code it's close to impossible to give you more
than hints.
I include some code which I wrote to peek into my running server with a
manhole connection. dumpobjects is loosely based on several Python
snippets and might not win a contest in programming but it helps to
track the number of objects of certain types in the running system when
used on the commandline from inside your server.
HTH, Werner
To create a manhole in the same process you started with your .tac file
use the following code:
---------------------------------------------------------------
def ManholeFactory(namespace, **passwords):
realm = manhole_ssh.TerminalRealm()
def getManhole(_):
return manhole.Manhole(namespace)
realm.chainedProtocolFactory.protocolFactory = getManhole
p = portal.Portal(realm)
p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords))
f = manhole_ssh.ConchFactory(p)
return f
console = ManholeFactory(globals(), admin='admin')
internet.TCPServer(2222, console,
interface='localhost').setServiceParent(application)
---------------------------------------------------------------
exc = [
"function",
"type",
"list",
"dict",
"tuple",
"wrapper_descriptor",
"module",
"method_descriptor",
"member_descriptor",
"instancemethod",
"builtin_function_or_method",
"frame",
"classmethod",
"classmethod_descriptor",
"_Environ",
"MemoryError",
"_Printer",
"_Helper",
"getset_descriptor",
"weakreaf"
]
inc = [
'YourObject_One',
'YourObject_Two'
]
prev = {}
def dumpObjects(delta=True, limit=0, include=inc, exclude=[]):
global prev
if include != [] and exclude != []:
print 'cannot use include and exclude at the same time'
return
print 'working with:'
print ' delta: ', delta
print ' limit: ', limit
print ' include: ', include
print ' exclude: ', exclude
objects = {}
gc.collect()
oo = gc.get_objects()
for o in oo:
if getattr(o, "__class__", None):
name = o.__class__.__name__
if ((exclude == [] and include == []) or \
(exclude != [] and name not in exclude) or \
(include != [] and name in include)):
objects[name] = objects.get(name, 0) + 1
## if more:
## print o
pk = prev.keys()
pk.sort()
names = objects.keys()
names.sort()
for name in names:
if limit == 0 or objects[name] > limit:
if not prev.has_key(name):
prev[name] = objects[name]
dt = objects[name] - prev[name]
if delta or dt != 0:
print '%0.6d -- %0.6d -- ' % (dt, objects[name]), name
prev[name] = objects[name]
Abdul-Wahid Paterson wrote:
Hi,
I am using Python 2.5.
I will try to use guppy as suggested in another post to so if it will
help. Otherwise yes, I will try to reduce it to the smallest possible
code to demonstrate the leak. At the moment the code is a bit complex so
is hard to track down.
Thanks for the help.
AW
On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <[email protected]
<mailto:[email protected]>> wrote:
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot
of Athena elements on my site and they are all working well from
a functional point of view. However, the Athena elements are
eating up RAM and as I leave things running the Twisted process
slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least
Python 2.5. Previous versions have garbage collection problems on
long-running apps like a Twisted web app.
Athena-based apps should not inherently leak memory, but adding
leaks of your own creation isn't too hard ;-)
If you are running 2.5 or later, you should try to reduce your app
to the smallest possible test version that demonstrates the leak,
and someone here can probably be of more help.
-phil
_______________________________________________
Twisted-web mailing list
[email protected] <mailto:[email protected]>
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
------------------------------------------------------------------------
_______________________________________________
Twisted-web mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________
Twisted-web mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
