On Feb 22, 2012, at 3:51 PM, Claudio Freire wrote: > On Wed, Feb 22, 2012 at 5:40 PM, Michael Bayer <mike...@zzzcomputing.com> > wrote: >> Saw that a bit, but looking at the "tips" at the bottom, concrete >> implementation changes are not coming to mind. An "eternal structure" is >> ubiquitous in any programming language. sys.modules is a big list of all >> the Python modules that have been imported, each one full of functions, >> classes, other data, these are all "eternal structures" - sys.modules is >> normally never cleaned out. I'm not seeing at what point you move beyond >> things that are in these modules into things that are so-called "eternal >> structures" that lead to inappropriate memory fragmentation. > > The thing to be careful about is when those eternal structures are created. > > If they're created at the beginning (as sys.modules, which is > populated with imports, which most of the time happen in the preamble > of .py files), then the resulting objects will have lower memory > locations and thus not get in the way. > > But if those structures are created after the program had time to fill > its address space with transient objects (say, lazy imports, caches), > then when the transient objects are deleted, the eternal structures > (with their high addresses) prevent the heap from shrinking. > > Such caches, for instance, are better made limited in lifespan (say, > giving them a finite lifetime, making them expire, actively cleaning > them from time to time). Structures that are truly required to be > eternal are better populated at load time, early in the program's > lifecycle. In my backend, for instance, queries are precompiled at > startup, to make sure they have lower memory addresses. This has > mostly solved SQLA-related memory fragmentation issues for me.
IMHO the whole point of using a high level, interpreted language like Python is that we don't have to be bogged down thinking like C programmers. How come I've never had a memory fragmentation issue before ? I've made "precompilation" an option for folks who really wanted it but I've never had a need for such a thing. And you can be sure I work on some very large and sprawling SQLAlchemy models these days. There are some caches here and there like the identifier cache as well as caches inside of TypeEngine objects, but these caches are all intended to be of limited size. -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
