gc.collect() was not called in the last sample. Here is one where gc.collect() is also called
<class 'sqlalchemy.util._collections.OrderedDict'>: last:131 curr:134 diff:3 <type 'builtin_function_or_method'>: last:862 curr:868 diff:6 <class 'sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2'>: last:18 curr:21 diff:3 <class 'sqlalchemy.types.NullType'>: last:19 curr:22 diff:3 <type 'instancemethod'>: last:561 curr:564 diff:3 <class 'sqlalchemy.types.Text'>: last:234 curr:264 diff:30 <class 'sqlalchemy.dialects.postgresql.psycopg2._PGArray'>: last:180 curr:210 diff:30 <class 'sqlalchemy.sql.expression._BindParamClause'>: last:585 curr:678 diff:93 <class 'sqlalchemy.util._collections.PopulateDict'>: last:18 curr:21 diff:3 <type 'list'>: last:2962 curr:2983 diff:21 <type 'instance'>: last:197 curr:200 diff:3 <class 'sqlalchemy.dialects.postgresql.base.PGTypeCompiler'>: last:18 curr:21 diff:3 <class 'sqlalchemy.dialects.postgresql.psycopg2.PGIdentifierPreparer_psycopg2'>: last:18 curr:21 diff:3 <type 'dict'>: last:10470 curr:10854 diff:384 <class 'sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2'>: last:18 curr:21 diff:3 <class 'sqlalchemy.types.String'>: last:493 curr:550 diff:57 <class 'sqlalchemy.types.DateTime'>: last:53 curr:56 diff:3 <type 'weakref'>: last:2165 curr:2297 diff:132 <type 'tuple'>: last:29338 curr:29473 diff:135 <type 'cell'>: last:933 curr:1023 diff:90 <class 'sqlalchemy.types.Integer'>: last:57 curr:63 diff:6 <type 'function'>: last:8139 curr:8262 diff:123 On Sunday, 13 January 2013 11:45:11 UTC+5:30, Michael Bayer wrote: > > > On Jan 13, 2013, at 12:40 AM, Anoop K wrote: > > Hi, > > I had POSTed a question( > https://groups.google.com/forum/?fromgroups=#!topic/sqlalchemy/vCjrGE9zYwQ > ) > related to session/engine cleanup. > > After fixing the engine cleanup. I could see that engine and its > associated objects are GCed properly. However > process memory is gradually increasing. I am creating and destroying > session/engine every 30 secs. > > SQLAlchemy usage: > > Every 30 sec following happens. > > *engine = create_engine(...)* > *sn = Session(bind=engine)...* > *try:* > * sn.merge* > * sn.commit* > *finally:* > * sn.close()* > * sn.bind.dispose()* > *>>> sn and engine don't have any reference after this point ....* > * > * > * > * > GC objects after some runs are listed below, Looks like some items are > increasing in memory(in BOLD). > *Is this some kind of leak ?* > > > First off, there's no way to know without a real, reproducing test case. > Providing this would give us something tangible to examine. > > As the pattern of creating and destroying engines many times is not the > primary use case for the Engine, it's hard to say definitively, though we > do have memory tests which test the pattern of engines being created and > disposed many times, at the same time testing the length of > gc.get_objects() to confirm no growth in memory. In your example below, > the objects in question are mostly type objects. These objects do have a > behavior in that as created, they will cache information about themselves > within a dialect that is local to an Engine. But as each Engine is > disposed, so is the Dialect, and so is the cache. The cache itself is > also a WeakKeyDictionary, so that no strong reference to the type object is > created. So the cache does not prevent the type objects from being garbage > collected. There are tests which ensure this behavior as well. > > In your counts below, there are 31 Dialect instances in one run and then > 38 in another. So this suggests that there are 31 and then 38 Engine > instances, a growth of 7, and that the Engine references are not in fact > being cleaned out, or gc.collect() is not being called within the test. > Depending on what you're doing with SQL expressions within each engine it > is plausible that a few hundred type objects have accumulated in the cache. > > Note that it is absolutely necessary to call gc.collect() in any Python > program that is attempting to get an accurate count of how many objects are > still strongly referenced, as unreachable cycles can remain for a > significant period of time if one waits for cyclic GC to happen > automatically. > > > > > <class 'sqlalchemy.util._collections.OrderedDict'>: last:176 curr:198 > diff:22 > <type 'builtin_function_or_method'>: last:900 curr:914 diff:14 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2'>: > last:31 curr:38 diff:7 > <class 'sqlalchemy.types.NullType'>: last:32 curr:39 diff:7 > <type 'instancemethod'>: last:605 curr:627 diff:22 > <class 'sqlalchemy.types.Text'>: last:177 curr:254 diff:77 > * <class 'sqlalchemy.dialects.postgresql.psycopg2._PGArray'>: last:120 > curr:197 diff:77* > *<class 'sqlalchemy.sql.expression._BindParamClause'>: last:658 curr:954 > diff:296* > <class 'sqlalchemy.util._collections.PopulateDict'>: last:60 curr:82 > diff:22 > <type 'list'>: last:3308 curr:3462 diff:154 > <type 'instance'>: last:139 curr:146 diff:7 > <class 'sqlalchemy.dialects.postgresql.base.PGTypeCompiler'>: last:31 > curr:38 diff:7 > <class > 'sqlalchemy.dialects.postgresql.psycopg2.PGIdentifierPreparer_psycopg2'>: > last:31 curr:38 diff:7 > <type 'dict'>: last:10930 curr:12172 diff:1242 > <class 'sqlalchemy.dialects.postgresql.psycopg2.PGCompiler_psycopg2'>: > last:60 curr:82 diff:22 > *<class 'sqlalchemy.types.String'>: last:571 curr:753 diff:182* > * <class 'sqlalchemy.types.DateTime'>: last:92 curr:113 diff:21* > <type 'weakref'>: last:2204 curr:2582 diff:378 > <type 'tuple'>: last:29850 curr:30231 diff:381 > <type 'cell'>: last:747 curr:978 diff:231 > <class 'sqlalchemy.types.Integer'>: last:52 curr:59 diff:7 > <type 'function'>: last:8077 curr:8392 diff:315 > > Thanks > Anoop > * > * > > > -- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/sqlalchemy/-/gESaFAd9DHkJ. > To post to this group, send email to [email protected]<javascript:> > . > To unsubscribe from this group, send email to > [email protected] <javascript:>. > For more options, visit this group at > http://groups.google.com/group/sqlalchemy?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/e3IWctBEpdsJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
