Re: [sqlalchemy] sqlalchemy gc and memory leak

2010-12-18 Thread Michael Bayer

On Dec 18, 2010, at 5:32 AM, drakkan wrote:

> Hi,
> 
> inspecting the gc I see a sqlalchemy memory leak in my application,
> here is the output from the gc:
> 
> : 2 -> 3 (+1)
> : 2 -> 3 (+1)
> : 1 -> 2 (+1)
> : 2 -> 3 (+1)
> : 5987 -> 5990 (+3)
> : 73 -> 80 (+7)
> : 8943 -> 8954 (+11)
> : 1 -> 2 (+1)
> : 180 -> 181 (+1)
> : 490 -> 487 (-3)
> : 2 -> 3 (+1)
> : 34 -> 35 (+1)
> : 3830 -> 3831 (+1)
> : 2953 -> 2963 (+10)
> 
> every time I make some database object a reference is added to
> sqlalchemy objects and never released. I'm sure the problem is my
> application and not sa, however I would like to know how to force
> sqlalchemy to delete objects references.
> 
> I'm using these function to query gc:
> 
> def gcHistogram(self):
>   import gc
>   result = {}
>   for o in gc.get_objects():
>   t = type(o)
>   count = result.get(t, 0)
>   result[t] = count + 1
>   print len(result)
>   return result
> 
>   def diffHists(self,h1, h2):
>   for k in h1:
>   if k in h2:
>   if h1[k] != h2[k]:
>   print "%s: %d -> %d (%s%d)" % (
>   k, h1[k], h2[k], h2[k] 
> > h1[k] and "+" or "", h2[k] - h1[k])

SQLA doesn't maintain strong references to anything at the module level.   The 
objects you have above appear to be related to the "compiled cache" used by an 
individual mapper.   This is an LRU-enabled dictionary that holds onto Insert 
constructs and their compiled form up to approximately 100 elements.   The 
dictionary is associated with a mapper, which in turn is associated with your 
mapped class.   If you dereference your mapped class (and all instances of that 
class), the mapper and the LRU cache will be gone.You can also say 
mapper._compiled_cache.clear() which will probably remove the 
Insert/SQLiteCompiler objects above.

As for Sessions, if you close(), commit(), rollback(), or expire_all() the 
session object, all user objects inside are weakly referenced by the session 
(though related objects will have strong references to each other).





> 
> thanks
> Nicola
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To post to this group, send email to sqlalch...@googlegroups.com.
> To unsubscribe from this group, send email to 
> sqlalchemy+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/sqlalchemy?hl=en.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to 
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/sqlalchemy?hl=en.



[sqlalchemy] sqlalchemy gc and memory leak

2010-12-18 Thread drakkan
Hi,

inspecting the gc I see a sqlalchemy memory leak in my application,
here is the output from the gc:

: 2 -> 3 (+1)
: 2 -> 3 (+1)
: 1 -> 2 (+1)
: 2 -> 3 (+1)
: 5987 -> 5990 (+3)
: 73 -> 80 (+7)
: 8943 -> 8954 (+11)
: 1 -> 2 (+1)
: 180 -> 181 (+1)
: 490 -> 487 (-3)
: 2 -> 3 (+1)
: 34 -> 35 (+1)
: 3830 -> 3831 (+1)
: 2953 -> 2963 (+10)

every time I make some database object a reference is added to
sqlalchemy objects and never released. I'm sure the problem is my
application and not sa, however I would like to know how to force
sqlalchemy to delete objects references.

I'm using these function to query gc:

def gcHistogram(self):
import gc
result = {}
for o in gc.get_objects():
t = type(o)
count = result.get(t, 0)
result[t] = count + 1
print len(result)
return result

def diffHists(self,h1, h2):
for k in h1:
if k in h2:
if h1[k] != h2[k]:
print "%s: %d -> %d (%s%d)" % (
k, h1[k], h2[k], h2[k] 
> h1[k] and "+" or "", h2[k] - h1[k])

thanks
Nicola

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to 
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/sqlalchemy?hl=en.