On Thu, May 17, 2012 at 10:21 AM, Michael Bayer <mike...@zzzcomputing.com> wrote: > There's a few different parts to what you're asking. > > The first is that you're comparing Python's use of OS memory (I'm assuming > this is the 200+ MB) to Python's actual amount of objects present. This is > a common mistake. Python up through version 2.6 does not release memory > back to the OS once taken - this was improved in 2.7. There's an old > article about this here: > http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm > as well as Alex Martelli's answer: http://stackoverflow.com/a/1316799/34549
I think you're mistaken about the version number in which that was changed. The article you pointed to (and my memory) seems to say it was on 2.5, not 2.7 > Second is, what exactly is the large object you're creating here ? Answer > - first, psycopg2 by default buffers the result set fully before returning > it to SQLAlchemy - so it is first a list of 5000 tuples. Second, the ORM > itself also by default buffers the full set of rows from the result set in > the form of mapped objects, so 5000 objects plus their related objects. A > way to modify this behavior is to use the yield_per() option of Query, which > will also in the case of psycopg2 tell psycopg2 to use its "server side > cursors" feature which does not buffer. It's important to notice that all that you mention on the py side would show up on heapy. But psycopg2's buffers would not. Nor would libpq's. > On May 17, 2012, at 7:33 AM, Vlad K. wrote: > The following is source of an example case. Requires SQLAlchemy (tested with > 0.7.5 and 0.7.7), guppy, psycopg2 (tested with 2.4.2 and 2.4.4). Happens > both on Fedora 15 64-bit and CentOS 6.2 32-bit, though of course the 32-bit > shows some 30% lower RES in top. Update to psycopg2 2.4.5, it fixes some memory leaks (they call them "reference counting problems"): http://initd.org/psycopg/articles/2012/03/29/psycopg-245-released/ Also... which version of libpq did you build psycopg2 with? > I don't see where and how would any objects remain in memory, and heapy > showing much lower memory use suggests something is retained in the involved > C extensions? C extensions, or memory fragmentation. If you read spanish, check this out: http://python.org.ar/pyar/Charlas#Depuraci.2BAPM-n_y_defragmentaci.2BAPM-n_de_memoria_en_Python If not... ehm... try google translate ;-) > Heapy before selecting rows: > Heapy after 5000 rows have been selected: Try the difference for more insight: h1 = hpy.heap() # do some h2 = hpy.heap() print repr(h2 - h1) -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
