On Sat, 22 Dec 2007 08:09:50 +0100, Fredrik Lundh wrote: > Steven D'Aprano wrote: > > > Not me. > > You're quite knew to this internet thing, aren't you? ;-)
:-D >> So... how do you measure memory usage in Python? Every programming >> language I've used before (not a huge range, I'll admit) had *some* >> sort of facility to measure memory usage, typically things like: >> >> * how much memory is free in the stack? >> >> * how much memory is free in the heap? >> >> * how big a block does this pointer point to? >> >> * how much memory does this record/struct/object/string use? > > And what languages would that be? I cannot think of a single modern > language that does any of that. Including low-level stuff like C/C++. I didn't actually say they were *modern* languages. E.g. THINK Pascal for Apple Mac, circa 1990. > And things like "how much memory is free in the heap" isn't even a > meaningful concept on a modern machine, thanks to the wonders of virtual > memory (especially the overcommitting kind). Maybe the memory model used in modern multi-tasking virtual-memory PCs makes the concept of "free memory" obsolete. Although if so, somebody should have a quiet word with the author of the Linux free command: $ free total used free shared buffers cached Mem: 1002524 988736 13788 0 7044 98916 -/+ buffers/cache: 882776 119748 Swap: 4241080 3939736 301344 (Admittedly that's system-wide memory usage, rather than for a single process.) > For Python, standard process monitoring tools (combined with a basic > understanding of how dynamic memory allocation works on modern > platforms) are usually sufficient to get a good view of an application's > memory usage patterns. Just run the program under a few different > scenarios, and see what it does. Are you saying that Python programs can't monitor their own memory use? I'm happy to accept that "free memory" is not a meaningful concept for a process in a modern system. That makes sense. But surely it is reasonable for a process to have an idea of how much memory it has actually used. Yes? No? > If the memory use looks suspicious, > use standard debugging techniques to locate the problematic area, and > standard benchmarking techniques to look for unexpected blowups and > leaks. What sort of "standard debugging techniques" work in the absence of any way (that I know of) to measure memory usage? In Python, standard debugging techniques usually start with the print statement, but one can't do anything like this: # problematic area of code last = memory() for i in xrange(100): x = foo() if memory() >= last: print "memory use increased", memory() So what are you suggesting is standard? > For really hard problems, use the "gc" module, instrumenting memory > allocation libraries (e.g. dmalloc), or C/C++-level debuggers. I'm not so much concerned about the really hard problems as I am about the really easy ones. -- Steven -- http://mail.python.org/mailman/listinfo/python-list