Werner, Thanks for your post. Is very useful.
I am using Guard so that might be one place to start looking although I personally didn't develop that part of the code. (Which kinda make me even more suspicious). The Manhole stuff looks interesting too. I will give that a go if I can't find anything by disabling the Guard code. Thanks again. AW On Tue, Dec 16, 2008 at 10:39 PM, Werner Thie <[email protected]>wrote: > Hi > > I was bitten by the same problem and went so far to track down all the > fishy cycles which easily can be created. Most of the cycles I detected were > multi-object cycles which were created by me just storing references to > objects to have them handy. Certain types of cycles cannot be broken by the > gc, you get a runaway situation either in Python24 or Python25. My code now > runs without problems for months serving more than 10k users a day providing > a card gaming environment. > > Things to check: > - the object hierarchy on the server and the client should be in sync, if > not detach might not work correctly > > - do you use stateful LivePages? > if so, check the toremember list in the request if its growing ad libidum > > - do you use guard/sessions? > if so, be careful with the session/mind object and what you store in mind > > But without seeing your code it's close to impossible to give you more than > hints. > > I include some code which I wrote to peek into my running server with a > manhole connection. dumpobjects is loosely based on several Python snippets > and might not win a contest in programming but it helps to track the number > of objects of certain types in the running system when used on the > commandline from inside your server. > > HTH, Werner > > To create a manhole in the same process you started with your .tac file use > the following code: > > --------------------------------------------------------------- > def ManholeFactory(namespace, **passwords): > realm = manhole_ssh.TerminalRealm() > > def getManhole(_): > return manhole.Manhole(namespace) > > realm.chainedProtocolFactory.protocolFactory = getManhole > p = portal.Portal(realm) > > > p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords)) > f = manhole_ssh.ConchFactory(p) > return f > > console = ManholeFactory(globals(), admin='admin') > internet.TCPServer(2222, console, > interface='localhost').setServiceParent(application) > --------------------------------------------------------------- > > exc = [ > "function", > "type", > "list", > "dict", > "tuple", > "wrapper_descriptor", > "module", > "method_descriptor", > "member_descriptor", > "instancemethod", > "builtin_function_or_method", > "frame", > "classmethod", > "classmethod_descriptor", > "_Environ", > "MemoryError", > "_Printer", > "_Helper", > "getset_descriptor", > "weakreaf" > ] > > inc = [ > 'YourObject_One', > 'YourObject_Two' > ] > > prev = {} > > def dumpObjects(delta=True, limit=0, include=inc, exclude=[]): > global prev > if include != [] and exclude != []: > print 'cannot use include and exclude at the same time' > return > print 'working with:' > print ' delta: ', delta > print ' limit: ', limit > print ' include: ', include > print ' exclude: ', exclude > objects = {} > gc.collect() > oo = gc.get_objects() > for o in oo: > if getattr(o, "__class__", None): > name = o.__class__.__name__ > if ((exclude == [] and include == []) or \ > (exclude != [] and name not in exclude) or \ > (include != [] and name in include)): > objects[name] = objects.get(name, 0) + 1 > ## if more: > ## print o > pk = prev.keys() > pk.sort() > names = objects.keys() > names.sort() > for name in names: > if limit == 0 or objects[name] > limit: > if not prev.has_key(name): > prev[name] = objects[name] > dt = objects[name] - prev[name] > if delta or dt != 0: > print '%0.6d -- %0.6d -- ' % (dt, objects[name]), name > prev[name] = objects[name] > > > > > > > > Abdul-Wahid Paterson wrote: > >> Hi, >> >> I am using Python 2.5. >> >> I will try to use guppy as suggested in another post to so if it will >> help. Otherwise yes, I will try to reduce it to the smallest possible code >> to demonstrate the leak. At the moment the code is a bit complex so is hard >> to track down. >> >> Thanks for the help. >> >> AW >> >> >> >> On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen >> <[email protected]<mailto: >> [email protected]>> wrote: >> >> On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote: >> >> I have managed to write my nevow/twisted site. I am using a lot >> of Athena elements on my site and they are all working well from >> a functional point of view. However, the Athena elements are >> eating up RAM and as I leave things running the Twisted process >> slowly consumes more and more RAM until the process dies. >> >> >> The first thing would be to check that you're running at least >> Python 2.5. Previous versions have garbage collection problems on >> long-running apps like a Twisted web app. >> >> Athena-based apps should not inherently leak memory, but adding >> leaks of your own creation isn't too hard ;-) >> >> If you are running 2.5 or later, you should try to reduce your app >> to the smallest possible test version that demonstrates the leak, >> and someone here can probably be of more help. >> >> -phil >> >> _______________________________________________ >> Twisted-web mailing list >> [email protected] <mailto:[email protected]> >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Twisted-web mailing list >> [email protected] >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web >> > > _______________________________________________ > Twisted-web mailing list > [email protected] > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web >
_______________________________________________ Twisted-web mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
