On Fri, Aug 15, 2014 at 11:26 AM, Narendra L feelna...@gmail.com wrote:
Actual question is here:
http://stackoverflow.com/questions/25200475/sqlalchmey-memory-leak-how-to-free-memory
What measures should I take to cleanup old session objects? Isn't
session.close() is sufficient?
or Is it something to do with pyramid?
Sqlalchmey setup:
--
def get_db(request):
maker = request.registry.dbmaker
session = maker()
@profile
def cleanup(request):
_session = request.db
if request.exception is not None:
_session.rollback()
else:
_session.commit()
_session.close()
# del _session # No memory released
request.add_finished_callback(cleanup)
return session
def main(global_config, **settings):
:
:
config.registry.dbmaker = sessionmaker(bind=engine)
config.add_request_method(get_db, name='db', reify=True)
:
:
Pyramid app request handler is like
@view_config(route_name='list_employees', renderer='json')
def employees(request):
session = request.db
office = session.query(Office).get(1)
employees = [x.name for x in office.employees]
return employees
Now the problem is, In every request to list_employees the memory is
growing. the size of increase in memory is almost equal to size of
office.employees.
Debug:
request 1 starts with memory utilization = 10MB
request 1 ends with memory utilization = 18MB
request 2 starts with memory utilization = 18MB
request 2 ends with memory utilization = 26MB
request 3 starts with memory utilization = 26MB
request 3 ends with memory utilization = 34MB
:
:
Grows eventually
employees = [x.name for x in office.employees]
This is the line where about 8-10MB memory utilized
To debug, I added __del__ method in Employ and Office models, looks like
they are deleting.
Also tried session.expunge(office), del office and gc.collect()
I am debugging memory consumption using
https://pypi.python.org/pypi/memory_profiler Also I am
usinghttps://pypi.python.org/pypi/transaction is other requests.
I have removed debug toolbar also
You might try adding dowser (https://pypi.python.org/pypi/dowser/0.2)
to your pyramid app - it gives you a way to find out what objects are
holding references to each other from your web browser. dowser is a
cherrypy app, but cherrypy supports WSGI so you can embed it in your
pyramid app something like this:
#
# In your configuration:
config.add_route('dowser', '/dowser/*subpath')
#
# In your views:
from pyramid.view import view_config
from pyramid.request import call_app_with_subpath_as_path_info
import dowser
import cherrypy
app = cherrypy.tree.mount(dowser.Root(), '/dowser/')
@view_config(route_name='dowser')
def dowserview(request):
request.subpath = request.matchdict['subpath']
return call_app_with_subpath_as_path_info(request, app)
#
Then visit http://host/dowser/ in your browser.
Inspecting SQLAlchemy objects is a bit tricky because of their
relationship with the session, which normally only lasts for the
lifetime of a single web request and is not thread safe. You'll
probably need to run your pyramid app in a single thread, and if
dowser tries to touch an attribute which would be lazy-loaded, it will
fail. But if you can work around those restrictions, it can be pretty
helpful.
(I've also got a memory leak in a pyramid/sqlalchemy application, and
I started trying to track it down using this method. So far I haven't
found the problem, and the application is not used very much, so it
hasn't been high on my priority list.)
Hope that helps,
Simon
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
