On Fri, Aug 15, 2014 at 11:26 AM, Narendra L <feelna...@gmail.com> wrote: > Actual question is here: > http://stackoverflow.com/questions/25200475/sqlalchmey-memory-leak-how-to-free-memory > > What measures should I take to cleanup old session objects? Isn't > session.close() is sufficient? > > or Is it something to do with pyramid? > > Sqlalchmey setup: > ---------------------------------------------------------------------------------- > def get_db(request): > maker = request.registry.dbmaker > session = maker() > > @profile > def cleanup(request): > _session = request.db > if request.exception is not None: > _session.rollback() > else: > _session.commit() > _session.close() > # del _session # No memory released > > request.add_finished_callback(cleanup) > return session > > def main(global_config, **settings): > : > : > config.registry.dbmaker = sessionmaker(bind=engine) > config.add_request_method(get_db, name='db', reify=True) > : > : > > Pyramid app request handler is like > > @view_config(route_name='list_employees', renderer='json') > def employees(request): > session = request.db > office = session.query(Office).get(1) > employees = [x.name for x in office.employees] > return employees > > Now the problem is, In every request to list_employees the memory is > growing. the size of increase in memory is almost equal to size of > office.employees. > > Debug: > > request 1 starts with memory utilization = 10MB > request 1 ends with memory utilization = 18MB > > request 2 starts with memory utilization = 18MB > request 2 ends with memory utilization = 26MB > > request 3 starts with memory utilization = 26MB > request 3 ends with memory utilization = 34MB > : > : > Grows eventually > > employees = [x.name for x in office.employees] > This is the line where about 8-10MB memory utilized > > To debug, I added __del__ method in Employ and Office models, looks like > they are deleting. > > Also tried session.expunge(office), del office and gc.collect() > > I am debugging memory consumption using > https://pypi.python.org/pypi/memory_profiler Also I am > usinghttps://pypi.python.org/pypi/transaction is other requests. > > I have removed debug toolbar also >
You might try adding dowser (https://pypi.python.org/pypi/dowser/0.2) to your pyramid app - it gives you a way to find out what objects are holding references to each other from your web browser. dowser is a cherrypy app, but cherrypy supports WSGI so you can embed it in your pyramid app something like this: #-------------------------------- # In your configuration: config.add_route('dowser', '/dowser/*subpath') #-------------------------------- # In your views: from pyramid.view import view_config from pyramid.request import call_app_with_subpath_as_path_info import dowser import cherrypy app = cherrypy.tree.mount(dowser.Root(), '/dowser/') @view_config(route_name='dowser') def dowserview(request): request.subpath = request.matchdict['subpath'] return call_app_with_subpath_as_path_info(request, app) #-------------------------------- Then visit http://<host>/dowser/ in your browser. Inspecting SQLAlchemy objects is a bit tricky because of their relationship with the session, which normally only lasts for the lifetime of a single web request and is not thread safe. You'll probably need to run your pyramid app in a single thread, and if dowser tries to touch an attribute which would be lazy-loaded, it will fail. But if you can work around those restrictions, it can be pretty helpful. (I've also got a memory leak in a pyramid/sqlalchemy application, and I started trying to track it down using this method. So far I haven't found the problem, and the application is not used very much, so it hasn't been high on my priority list.) Hope that helps, Simon -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
