Re: [sqlalchemy] Understanding memory usage under SQLA

2012-06-21 Thread Vlad K.
On 05/17/2012 05:09 PM, Claudio Freire wrote: Precompiling queries in SQLA, to populate the various SQLA's compiler caches, doing some queries that cause libpq and psycopg2 to excercise (and thus to allocate whatever permanent data structures it needs to), all at load time, will help keep

[sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Vlad K.
Hello. I have a problem when processing relatively large number of rows. For example, when selecting 5000 main rows, each having a number of many-to-one relationships, memory usage shown by top skyrockets into 200+ MB range (RES), while heapy shows cca 20MB of

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Michael Bayer
There's a few different parts to what you're asking. The first is that you're comparing Python's use of OS memory (I'm assuming this is the 200+ MB) to Python's actual amount of objects present. This is a common mistake. Python up through version 2.6 does not release memory back to the OS

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Claudio Freire
On Thu, May 17, 2012 at 10:21 AM, Michael Bayer mike...@zzzcomputing.com wrote: There's a few different parts to what you're asking. The first is that you're comparing Python's use of OS memory (I'm assuming this is the 200+ MB) to Python's actual amount of objects present.   This is a common

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Vlad K.
Hi, thanks for your reply. Yes, I know the Python memory management model and that while it may free internally, it does not to OS. I've read somewhere that it has been fixed in 2.7 as well, but my testing on Fedora's 2.7.1 still shows the issue. That's why I thought perhaps there are

Re: [sqlalchemy] Understanding memory usage under SQLA

2012-05-17 Thread Claudio Freire
On Thu, May 17, 2012 at 11:57 AM, Vlad K. v...@haronmedia.com wrote: PostgreSQL 9.0.7, libpq is part of the same version -devel package PostgreSQL 9.1.3 (via pgsql yum repo), libpq is part of the same version -devel package 9.1 (the one you have in production) also has leak-related fixes. They