On Wed, Feb 22, 2012 at 5:40 PM, Michael Bayer <mike...@zzzcomputing.com> wrote: > Saw that a bit, but looking at the "tips" at the bottom, concrete > implementation changes are not coming to mind. An "eternal structure" is > ubiquitous in any programming language. sys.modules is a big list of all the > Python modules that have been imported, each one full of functions, classes, > other data, these are all "eternal structures" - sys.modules is normally > never cleaned out. I'm not seeing at what point you move beyond things > that are in these modules into things that are so-called "eternal structures" > that lead to inappropriate memory fragmentation.
The thing to be careful about is when those eternal structures are created. If they're created at the beginning (as sys.modules, which is populated with imports, which most of the time happen in the preamble of .py files), then the resulting objects will have lower memory locations and thus not get in the way. But if those structures are created after the program had time to fill its address space with transient objects (say, lazy imports, caches), then when the transient objects are deleted, the eternal structures (with their high addresses) prevent the heap from shrinking. Such caches, for instance, are better made limited in lifespan (say, giving them a finite lifetime, making them expire, actively cleaning them from time to time). Structures that are truly required to be eternal are better populated at load time, early in the program's lifecycle. In my backend, for instance, queries are precompiled at startup, to make sure they have lower memory addresses. This has mostly solved SQLA-related memory fragmentation issues for me. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
