[sqlalchemy] Re: Memory leak - is session.close() sufficient?
On Nov 10, 2008, at 9:05 AM, Julien Cigar wrote: > > One solution is to use server side cursors, but it only works with > PostgreSQL at the moment (and you can't use server side cursors with > the > ORM). you can. set server_side_cursors=True on your create_engine() call when using the postgres dialect. Also cx_oracle cursors are inherently "server side". --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
Note that a lot of database drivers cache *everything* in memory when you .fetchall(), fetchone() or fetchmany(x). So all those operations consume the same amout of memory : result = cursor.execute(...) for i in result: ... data = result.fetchall() for i in data: ... data = result.fetchone() ... With a lot of rows it can consume a lot of memory ... One solution is to use server side cursors, but it only works with PostgreSQL at the moment (and you can't use server side cursors with the ORM). On Mon, 2008-11-10 at 05:47 -0800, joelanman wrote: > Thanks Simon - just checked and I'm running 2.5.2 on my machines. > > From experimenting - I'm not so sure I have a memory leak, so much as > just using a lot of memory. > > I didn't realise that when Python frees memory, it doesnt necessarily > become free in Linux. I think that possibly all that's happening is > that as more complex pages are hit, the app is using more memory - > which is never obviously freed up in Linux. > > On that note then - for simpler pages, the Apache process seem to use > about 60mb, which goes up to around 150mb for more complex pages > (Joined objects made up of around 3500 rows from mysql). > > I'm using WSGIDaemonProcess with 8 threads - so do these figures sound > like an alright ballpark or totally ridiculous? Is there anything I > can do to keep memory usage down? > > -- Julien Cigar Belgian Biodiversity Platform http://www.biodiversity.be Université Libre de Bruxelles (ULB) Campus de la Plaine CP 257 Bâtiment NO, Bureau 4 N4 115C (Niveau 4) Boulevard du Triomphe, entrée ULB 2 B-1050 Bruxelles Mail: [EMAIL PROTECTED] @biobel: http://biobel.biodiversity.be/person/show/471 Tel : 02 650 57 52 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
Thanks Simon - just checked and I'm running 2.5.2 on my machines. >From experimenting - I'm not so sure I have a memory leak, so much as just using a lot of memory. I didn't realise that when Python frees memory, it doesnt necessarily become free in Linux. I think that possibly all that's happening is that as more complex pages are hit, the app is using more memory - which is never obviously freed up in Linux. On that note then - for simpler pages, the Apache process seem to use about 60mb, which goes up to around 150mb for more complex pages (Joined objects made up of around 3500 rows from mysql). I'm using WSGIDaemonProcess with 8 threads - so do these figures sound like an alright ballpark or totally ridiculous? Is there anything I can do to keep memory usage down? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
> -Original Message- > From: sqlalchemy@googlegroups.com > [mailto:[EMAIL PROTECTED] On Behalf Of joelanman > Sent: 10 November 2008 00:21 > To: sqlalchemy > Subject: [sqlalchemy] Re: Memory leak - is session.close() sufficient? > > > Thanks for all the advice - I've changed my unicode settings and > upgraded Beaker, but still have something to fix.. I'll report back if > I find it. I'm sure it's not relevant, but for a while I was developing an application with SQLAlchemy (0.3.something) on Python 2.4.1, and had problems with memory leaks. I upgraded to Python 2.4.4 and the leaks went away. Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
Thanks for all the advice - I've changed my unicode settings and upgraded Beaker, but still have something to fix.. I'll report back if I find it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
On Nov 7, 2008, at 10:17 AM, joelanman wrote: > > Thanks for that - I'll check out those options - I am using Beaker for > cache and sessions. OK, with Beaker, if you are caching things on a dynamically generated key, such as a key constructed from arbitrary parameters, I strongly recommend you work with Beaker 1.1 which is not yet released, but is available from mercurial. Previous versions will store an in-memory record for each unique key, and if you have an arbitrary number of keys then you'll have an arbitrary growth in memory. 1.1 has a significant rearrangement of things so that this problem is resolved. Try disabling Beaker entirely to see if that clears the issue. > In the meantime I found this post about a leak in MySQLdb 1.2.2 when > using charset=UTF8, which I am: > > http://jjinux.blogspot.com/2008/09/python-debugging-memory-leaks.html > > I'm using SQLA to do all database activity - would this leak be > relevant? I tried out his test program there and indeed it leaks memory like a wellspring, even if you put a cursor.close() in there. The good news is, when using SQLAlchemy with MySQL, you want to connect using ? charset=utf8&use_unicode=0 , which disables MySQLdb's conversion of python bytestrings into unicode objects, since SQLA handles this in a more finely-controllable way (not to mention, non-leaking :) ). When I add use_unicode=0 to the test script, the memory leak ends. > Also - if you have time, could you explain your point about __del__? > If I don't define it and call session.close(), will the session close > itself anyway when the controller that called it expires? The session will actually result in everything being closed out if it just falls out of scope. However, this might place some burden on the garbage collector to take care of it and wont necessarily close things out immediately. I don't know what web.py's architecture offers, but usually session management is assembled into a web framework by wrapping requests with a session open/close block, so that its handled in all cases without reliance on gc. This is described at http://www.sqlalchemy.org/docs/05/session.html#unitofwork_contextual_lifespan . --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
Thanks for that - I'll check out those options - I am using Beaker for cache and sessions. In the meantime I found this post about a leak in MySQLdb 1.2.2 when using charset=UTF8, which I am: http://jjinux.blogspot.com/2008/09/python-debugging-memory-leaks.html I'm using SQLA to do all database activity - would this leak be relevant? Also - if you have time, could you explain your point about __del__? If I don't define it and call session.close(), will the session close itself anyway when the controller that called it expires? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Memory leak - is session.close() sufficient?
On Nov 7, 2008, at 6:21 AM, joelanman wrote: > > Hi, > > I'm getting a memory leak with my app - the stack is apache2, > mod_wsgi, web.py - and then a lot of sqlalchemy and my own code. The > issue may well not be with my usage of SQLA - just making sure there's > nothing I might be doing wrong with it. > > At the start of every web request (__init__ for a controller) I open a > Session(), and in __del__ I close() it. Is there anything else I ought > to do? that is the correct procedure. The Session weak references everything anyway so its hard for it to cause a fast leak. > Any pointers on how to chase down my memory problem? one way is to visually inspect your application for any global variables, like dictionaries, caches, etc. and to analyze if they grow arbitrarily. Anything that is global or held across requests is suspect and should be analyzed carefully. Also, any usage of __del__ should be carefully scrutinized, as a cyclical reference on an object that implements __del__ cannot be garbage collected (that you're using __del__ might be the issue here. __del__ is actually never really needed as you can use a weakref callback for cleanup activities). Some third party libraries can also cause "leaky" behavior, such as Beaker. Try disabling any caching utilities to see if that isolates the issue. The other way is to analyze leaks is to use gc facilities - such as peeking into gc.get_objects() to see what kinds of objects are growing the collection, or gc.set_debug(gc.DEBUG_LEAK) which deals with unreachable objects (i.e. because of __del__).I usually opt for the former since I don't really use __del__ but I use global caches of things quite a bit. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
