[sqlalchemy] Sqlalchmey session memory is never released when used with pyramid.
Actual question is here: http://stackoverflow.com/questions/25200475/sqlalchmey-memory-leak-how-to-free-memory What measures should I take to cleanup old session objects? Isn't session.close() is sufficient? or Is it something to do with pyramid? Sqlalchmey setup:--def get_db(request): maker = request.registry.dbmaker session = maker() @profile def cleanup(request): _session = request.db if request.exception is not None: _session.rollback() else: _session.commit() _session.close() # del _session # No memory released request.add_finished_callback(cleanup) return session def main(global_config, **settings): : : config.registry.dbmaker = sessionmaker(bind=engine) config.add_request_method(get_db, name='db', reify=True) : : Pyramid app request handler is like @view_config(route_name='list_employees', renderer='json')def employees(request): session = request.db office = session.query(Office).get(1) employees = [x.name for x in office.employees] return employees Now the problem is, In every request to list_employees the memory is growing. the size of increase in memory is almost equal to size of office.employees. Debug: request 1 starts with memory utilization = 10MB request 1 ends with memory utilization = 18MB request 2 starts with memory utilization = 18MB request 2 ends with memory utilization = 26MB request 3 starts with memory utilization = 26MB request 3 ends with memory utilization = 34MB : : Grows eventually employees = [x.name for x in office.employees]This is the line where about 8-10MB memory utilized To debug, I added *__del__* method in Employ and Office models, looks like they are deleting. Also tried session.expunge(office), del office and gc.collect() I am debugging memory consumption using https://pypi.python.org/pypi/memory_profiler Also I am using https://pypi.python.org/pypi/transaction is other requests. I have removed debug toolbar also -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Sqlalchmey session memory is never released when used with pyramid.
On Fri, Aug 15, 2014 at 11:26 AM, Narendra L feelna...@gmail.com wrote:
Actual question is here:
http://stackoverflow.com/questions/25200475/sqlalchmey-memory-leak-how-to-free-memory
What measures should I take to cleanup old session objects? Isn't
session.close() is sufficient?
or Is it something to do with pyramid?
Sqlalchmey setup:
--
def get_db(request):
maker = request.registry.dbmaker
session = maker()
@profile
def cleanup(request):
_session = request.db
if request.exception is not None:
_session.rollback()
else:
_session.commit()
_session.close()
# del _session # No memory released
request.add_finished_callback(cleanup)
return session
def main(global_config, **settings):
:
:
config.registry.dbmaker = sessionmaker(bind=engine)
config.add_request_method(get_db, name='db', reify=True)
:
:
Pyramid app request handler is like
@view_config(route_name='list_employees', renderer='json')
def employees(request):
session = request.db
office = session.query(Office).get(1)
employees = [x.name for x in office.employees]
return employees
Now the problem is, In every request to list_employees the memory is
growing. the size of increase in memory is almost equal to size of
office.employees.
Debug:
request 1 starts with memory utilization = 10MB
request 1 ends with memory utilization = 18MB
request 2 starts with memory utilization = 18MB
request 2 ends with memory utilization = 26MB
request 3 starts with memory utilization = 26MB
request 3 ends with memory utilization = 34MB
:
:
Grows eventually
employees = [x.name for x in office.employees]
This is the line where about 8-10MB memory utilized
To debug, I added __del__ method in Employ and Office models, looks like
they are deleting.
Also tried session.expunge(office), del office and gc.collect()
I am debugging memory consumption using
https://pypi.python.org/pypi/memory_profiler Also I am
usinghttps://pypi.python.org/pypi/transaction is other requests.
I have removed debug toolbar also
You might try adding dowser (https://pypi.python.org/pypi/dowser/0.2)
to your pyramid app - it gives you a way to find out what objects are
holding references to each other from your web browser. dowser is a
cherrypy app, but cherrypy supports WSGI so you can embed it in your
pyramid app something like this:
#
# In your configuration:
config.add_route('dowser', '/dowser/*subpath')
#
# In your views:
from pyramid.view import view_config
from pyramid.request import call_app_with_subpath_as_path_info
import dowser
import cherrypy
app = cherrypy.tree.mount(dowser.Root(), '/dowser/')
@view_config(route_name='dowser')
def dowserview(request):
request.subpath = request.matchdict['subpath']
return call_app_with_subpath_as_path_info(request, app)
#
Then visit http://host/dowser/ in your browser.
Inspecting SQLAlchemy objects is a bit tricky because of their
relationship with the session, which normally only lasts for the
lifetime of a single web request and is not thread safe. You'll
probably need to run your pyramid app in a single thread, and if
dowser tries to touch an attribute which would be lazy-loaded, it will
fail. But if you can work around those restrictions, it can be pretty
helpful.
(I've also got a memory leak in a pyramid/sqlalchemy application, and
I started trying to track it down using this method. So far I haven't
found the problem, and the application is not used very much, so it
hasn't been high on my priority list.)
Hope that helps,
Simon
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Re: Sqlalchmey session memory is never released when used with pyramid.
I'd suggest two things: 1- Post the OS and Python versions 2- Build a minimal self-contained version of the issue, that you can share on github Aside from making it easier to identify the issue yourself, there have been a handful of odd memory leaks on Pyramid that are specific to particular OS+Python combinations. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] cannot access tables
On Aug 15, 2014, at 5:03 PM, Greg Silverman g...@umn.edu wrote:
Then, I thought, what if this is an SQLAlchemy issue. Looks to be. I ran the
following script as a test:
import pyodbc
import sqlalchemy
from sqlalchemy.engine import reflection
from sqlalchemy.engine.reflection import Inspector
def connect():
return pyodbc.connect(
'DRIVER={FreeTDS};SERVER=ip_address;'
'DATABASE=db_name;UID=test;PWD=test;port=1433;'
'TDS_Version=9.1;')
engine = sqlalchemy.create_engine('mssql://', creator=connect)
conn = engine.connect()
print conn
for row in engine.execute('select 6 * 7 as [Result];'):
print row.Result
insp = reflection.Inspector.from_engine(engine)
table_name = 'irb_desc'
table_names = insp.get_table_names()
if table_name not in table_names:
print 'A'
Again, I am connecting fine with the database create.engine method (that is
'42' is printing as expected), but when I run the inspector.get_table_names
method with the given conditional it is printing the 'A' (I have tried other
table names in the same database to which I added 'irbd_balance,' all with
the same result.
what is the SQL output if you set echo='debug'; then, take the SQL you see
and take a look at what it's SELECTing so you can see what might be wrong.
Probably some schema name setting or something like that.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
