Re: [sqlalchemy] Re: sqlalchemy gc and memory leak
On Dec 18, 2010, at 9:56 AM, drakkan wrote:
> Thanks for your response Michael,
>
> after some other tests I can see some objects released:
>
> : 239 -> 199 (-40)
> : 630 -> 531 (-99)
> : 147 -> 107 (-40)
> : 98 -> 28 (-70)
> : 84 -> 24 (-60)
> : 3851 -> 3825 (-26)
> : 14 -> 4 (-10)
> : 7 -> 2 (-5)
> : 42 -> 12 (-30)
> : 14 -> 4 (-10)
> : 234 -> 96 (-138)
> : 799 -> 464 (-335)
> : 175 -> 115
> (-60)
> : 2390 -> 1370 (-1020)
> : 542 -> 277
> (-265)
> : 11 -> 8 (-3)
> : 6 -> 3 (-3)
> : 6 -> 3 (-3)
> : 14 -> 4 (-10)
> : 14 -> 4 (-10)
> : 11147 -> 11142 (-5)
> : 7 -> 2 (-5)
> : 14 -> 4 (-10)
> : 49 -> 14 (-35)
> : 45 -> 25 (-20)
> : 7 -> 2 (-5)
> : 14 -> 4 (-10)
> : 619 -> 299 (-320)
> : 4208 -> 3304 (-904)
> : 24 -> 7 (-17)
> : 7 -> 2 (-5)
> : 7058 -> 6291 (-767)
> : 86 -> 46 (-40)
> : 15 -> 5 (-10)
> : 10985 -> 9474 (-1511)
> : 78 -> 68 (-10)
> : 42 -> 12 (-30)
> : 10 -> 3
> (-7)
> : 336 -> 331 (-5)
> : 7 -> 2 (-5)
> : 42 -> 12 (-30)
> : 87 -> 37 (-50)
> : 60 -> 22 (-38)
>
>
> for example explorsrv.models.samodels.ConfigurazioneQuickCall is a
> related object and after some database activity 7 references were
> collected, as you can see from the gc graph above.
>
> after each database task I call a method such this:
>
> @staticmethod
> def tryCommitAndClose(session,ritorno=None):
>try:
> session.commit()
>except:
> session.rollback()
> session.close()
> return ritorno
>else:
> session.close()
> return True
>
> and this should release the session resources, so in summary my
> sqlalchemy usage should be safe and even if the memory isn't
> immediatly released it should not leak, I understand you correctly?
When we do memory tests we typically take an application loop we'd like to test
and then run it about 50 times, counting the total size of gc, and we then look
to make sure that gc isn't growing over time.A "sawtooth" pattern in gc
growth is common - meaning gc grows for 6-10 iterations, then it goes down
again. This is actually in some cases indpendent of calls to gc.collect().
Newer versions of Pysqlite for example have some internal references that are
cleared out periodically which cause this symptom to appear. If you want to
try our test fixture the profile function is at
http://www.sqlalchemy.org/trac/browser/test/aaa_profiling/test_memusage.py#L32 .
>
> On 18 Dic, 15:02, Michael Bayer wrote:
>> On Dec 18, 2010, at 5:32 AM, drakkan wrote:
>>
>>
>>
>>> Hi,
>>
>>> inspecting the gc I see a sqlalchemy memory leak in my application,
>>> here is the output from the gc:
>>
>>> : 2 -> 3 (+1)
>>> : 2 -> 3 (+1)
>>> : 1 -> 2 (+1)
>>> : 2 -> 3 (+1)
>>> : 5987 -> 5990 (+3)
>>> : 73 -> 80 (+7)
>>> : 8943 -> 8954 (+11)
>>> : 1 -> 2 (+1)
>>> : 180 -> 181 (+1)
>>> : 490 -> 487 (-3)
>>> : 2 -> 3 (+1)
>>> : 34 -> 35 (+1)
>>> : 3830 -> 3831 (+1)
>>> : 2953 -> 2963 (+10)
>>
>>> every time I make some database object a reference is added to
>>> sqlalchemy objects and never released. I'm sure the problem is my
>>> application and not sa, however I would like to know how to force
>>> sqlalchemy to delete objects references.
>>
>>> I'm using these function to query gc:
>>
>>> def gcHistogram(self):
>>>import gc
>>>result = {}
>>>for o in gc.get_objects():
>>>t = type(o)
>>>count = result.get(t, 0)
>>>result[t] = count + 1
>>>print len(result)
>>>return result
>>
>>>def diffHists(self,h1, h2):
>>>for k in h1:
>>>if k in h2:
>>>if h1[k] != h2[k]:
>>>print "%s: %d -> %d (%s%d)" % (
>>>k, h1[k], h2[k], h2[k] >
>>> h1[k] and "+" or "", h2[k] - h1[k])
>>
>> SQLA doesn't maintain strong references to anything at the module level.
>> The objects you have above appear to be related to the "compiled cache" used
>> by an individual mapper. This is an LRU-enabled dictionary that holds onto
>> Insert constructs and their compiled form up to approximately 100 elements.
>> The dictionary is associated with a mapper, which in turn is associated
>> with your mapped class. If you dereference your mapped class (and all
>> instances of that class), the mapper and the LRU cache will be gone.You
>> can also say mapper._compiled_cache.clear() which will probably remove the
>> Insert/SQLiteCompiler objects above.
>>
>> As for Sessions, if you close(), commit(), rollback(), or expire_all() the
>> session object, all user objects inside are weakly referenced by the session
>> (though related objects will have strong references to each other).
>>
>>
>>
>>> thanks
>>> Nicola
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "sqlalchemy" group.
>>> To post to this group, send email to sqlalch...@googlegroups.com.
>>> To un
[sqlalchemy] Re: sqlalchemy gc and memory leak
Thanks for your response Michael,
after some other tests I can see some objects released:
: 239 -> 199 (-40)
: 630 -> 531 (-99)
: 147 -> 107 (-40)
: 98 -> 28 (-70)
: 84 -> 24 (-60)
: 3851 -> 3825 (-26)
: 14 -> 4 (-10)
: 7 -> 2 (-5)
: 42 -> 12 (-30)
: 14 -> 4 (-10)
: 234 -> 96 (-138)
: 799 -> 464 (-335)
: 175 -> 115
(-60)
: 2390 -> 1370 (-1020)
: 542 -> 277
(-265)
: 11 -> 8 (-3)
: 6 -> 3 (-3)
: 6 -> 3 (-3)
: 14 -> 4 (-10)
: 14 -> 4 (-10)
: 11147 -> 11142 (-5)
: 7 -> 2 (-5)
: 14 -> 4 (-10)
: 49 -> 14 (-35)
: 45 -> 25 (-20)
: 7 -> 2 (-5)
: 14 -> 4 (-10)
: 619 -> 299 (-320)
: 4208 -> 3304 (-904)
: 24 -> 7 (-17)
: 7 -> 2 (-5)
: 7058 -> 6291 (-767)
: 86 -> 46 (-40)
: 15 -> 5 (-10)
: 10985 -> 9474 (-1511)
: 78 -> 68 (-10)
: 42 -> 12 (-30)
: 10 -> 3
(-7)
: 336 -> 331 (-5)
: 7 -> 2 (-5)
: 42 -> 12 (-30)
: 87 -> 37 (-50)
: 60 -> 22 (-38)
for example explorsrv.models.samodels.ConfigurazioneQuickCall is a
related object and after some database activity 7 references were
collected, as you can see from the gc graph above.
after each database task I call a method such this:
@staticmethod
def tryCommitAndClose(session,ritorno=None):
try:
session.commit()
except:
session.rollback()
session.close()
return ritorno
else:
session.close()
return True
and this should release the session resources, so in summary my
sqlalchemy usage should be safe and even if the memory isn't
immediatly released it should not leak, I understand you correctly?
On 18 Dic, 15:02, Michael Bayer wrote:
> On Dec 18, 2010, at 5:32 AM, drakkan wrote:
>
>
>
> > Hi,
>
> > inspecting the gc I see a sqlalchemy memory leak in my application,
> > here is the output from the gc:
>
> > : 2 -> 3 (+1)
> > : 2 -> 3 (+1)
> > : 1 -> 2 (+1)
> > : 2 -> 3 (+1)
> > : 5987 -> 5990 (+3)
> > : 73 -> 80 (+7)
> > : 8943 -> 8954 (+11)
> > : 1 -> 2 (+1)
> > : 180 -> 181 (+1)
> > : 490 -> 487 (-3)
> > : 2 -> 3 (+1)
> > : 34 -> 35 (+1)
> > : 3830 -> 3831 (+1)
> > : 2953 -> 2963 (+10)
>
> > every time I make some database object a reference is added to
> > sqlalchemy objects and never released. I'm sure the problem is my
> > application and not sa, however I would like to know how to force
> > sqlalchemy to delete objects references.
>
> > I'm using these function to query gc:
>
> > def gcHistogram(self):
> > import gc
> > result = {}
> > for o in gc.get_objects():
> > t = type(o)
> > count = result.get(t, 0)
> > result[t] = count + 1
> > print len(result)
> > return result
>
> > def diffHists(self,h1, h2):
> > for k in h1:
> > if k in h2:
> > if h1[k] != h2[k]:
> > print "%s: %d -> %d (%s%d)" % (
> > k, h1[k], h2[k], h2[k] >
> > h1[k] and "+" or "", h2[k] - h1[k])
>
> SQLA doesn't maintain strong references to anything at the module level.
> The objects you have above appear to be related to the "compiled cache" used
> by an individual mapper. This is an LRU-enabled dictionary that holds onto
> Insert constructs and their compiled form up to approximately 100 elements.
> The dictionary is associated with a mapper, which in turn is associated with
> your mapped class. If you dereference your mapped class (and all instances
> of that class), the mapper and the LRU cache will be gone. You can also
> say mapper._compiled_cache.clear() which will probably remove the
> Insert/SQLiteCompiler objects above.
>
> As for Sessions, if you close(), commit(), rollback(), or expire_all() the
> session object, all user objects inside are weakly referenced by the session
> (though related objects will have strong references to each other).
>
>
>
> > thanks
> > Nicola
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "sqlalchemy" group.
> > To post to this group, send email to sqlalch...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > sqlalchemy+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/sqlalchemy?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
