Re: Python not giving free memory back to the os get's me in real problems ...
You might try looking at references between objects, especially if there are any cyclic refs. For instance, if you have a data structure in which child nodes have back refs to their parents, try changing these to use weakref's. This may help the garbage collector to better reclaim discarded objects, so that not so many will need to be held in memory. -- Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
Diez B. Roggisch [EMAIL PROTECTED] wrote: Maybe delegating the actual processing to an external python process you feed through the macro might work. Thanks for the idea. I'll check it out soon and will report about possible improvements. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
I'm a bit fuzzy on this, but I don't think there _is_ a practical way to return memory to the OS in many OSes. That's not true at all. Most C libraries these days manage to return memory to the operating system. On Win32 (which the OP is most likely to use), the operating system offers the VirtualFree function to release memory to the operating system. In MSVCRT, the small block allocator will invoke VirtualFree inside __sbh_free_block, when a block group becomes completely free on the C level; likewise, _heap_free_region invokes VirtualFree, so does _heapmin_region. In glibc 2 on Linux, public_fREe will invoke munmap to release the memory to the operating system if the chunk being freed was mmap'ed. glibc allocates chunks greater than mmap_threshold with mmap, which defaults to 128kiB - so Python's obmalloc arenas fall under that algorithm. For small blocks, glibc releases topmost memory through sbrk when it gets more than M_TRIM_THRESHOLD bytes (defaults to 128kiB as well). This algorithm isn't used when MORECORE_CANNOT_TRIM is defined, which it would only on systems where sbrk does not support negative arguments. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
The moment I close the application that launched the macro, my ressources get freed. So is there a way to free my memory inside my nested loops? Yes. Most likely, it is your algorithm itself which is flawed (not Python, not the application that embeds Python): You keep, most likely, references to objects which you should release in order to be able to reuse the memory. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
[EMAIL PROTECTED] wrote: Our (python-)macro uses massively nested loops which are unfortunately necessary. These loops perform complex calculations in this commercial tool. To give you a quick overview how long this macros runs: The outer loop takes 5-7 hours for one cycle. Each cycle creates one outputfile. So we would like to perform 3-5 outer cycles en bloc. Unfortunately one of our computers (768MB RAM) crashes after just ~10% of the first cycle with the following error message: http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg while another computer (1GB RAM) crashes after ~10% of the fourth loop. While the virtual memory on the 1gb machine was full to the limit when it crashed the memory usage of the 768mb machine looked this this: While Python won't return memory to the OS, this is not the same as a memory leak. When your Python program is done using some objects, the memory they needed can be used by other Python objects in the same process. The process size should never grow bigger than the actually needed memory at a point in time. The (not only) Python problem is that the virtual memory of your process will never shrink, once it has attained its maximum size. Since the unused parts of its virtual memory will be paged out to disk, this is probably not big a problem anyway. If you have a long running process that you want to keep slim, it might be worth factoring out the memory intensive parts to separate processes like this in your macros: for x,y,z in outer_loop_sequence: os.system('python the_real_worker.py %s %s %s' % (x,y,z)) This should contain the big memory usage to short-running processes. It should not solve your crash problem though. If your crashes go away if you rig it like this, something was broken, either in your macro, or in the way the application uses Python for macros. You won't spend less memory because you divide the problem to two processes. To actually solve the problem, you need to look closer at your algorithms and data structures. Are you using effective ways of storing data? For instance, I imagine that numarray or whatever those things are called these days are much, much more effective at handling a vector of numerical values than a Python list of e.g. ints. Lists are pretty expensive. Lists of ints are at least twice as big as efficient integer arrays. Are you copying data when you should just share references? Are you hanging on to data longer than you actually need it? I don't see why there would be a big difference if you use Python or e.g. C or C++ to solve this kind of problem. If you use lots of data, you need to understand the data structures and use them effectively. With Python you are get shorter development time and less code to maintain to solve a certain problem. The price for that is typically longer execution time, but disregarding small systems that can't bear the size of the Python runtime, memory should not be a big problem. Python isn't Java. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
On 2007-04-26, Martin v. Löwis [EMAIL PROTECTED] wrote: I'm a bit fuzzy on this, but I don't think there _is_ a practical way to return memory to the OS in many OSes. That's not true at all. Most C libraries these days manage to return memory to the operating system. [...] Thanks for providing a real answer. Like I said in my post, my information wasn't very current. -- Grant Edwards grante Yow! Uh-oh!! I'm having at TOO MUCH FUN!! visi.com -- http://mail.python.org/mailman/listinfo/python-list
Python not giving free memory back to the os get's me in real problems ...
So I read quite a few things about this phenomenon in Python 2.4.x but I can hardly believe that there is really no solution to my problem. We use a commercial tool that has a macro functionality. These macros are written in python. So far nothing extraordinary. Our (python-)macro uses massively nested loops which are unfortunately necessary. These loops perform complex calculations in this commercial tool. To give you a quick overview how long this macros runs: The outer loop takes 5-7 hours for one cycle. Each cycle creates one outputfile. So we would like to perform 3-5 outer cycles en bloc. Unfortunately one of our computers (768MB RAM) crashes after just ~10% of the first cycle with the following error message: http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg while another computer (1GB RAM) crashes after ~10% of the fourth loop. While the virtual memory on the 1gb machine was full to the limit when it crashed the memory usage of the 768mb machine looked this this: http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg The moment I close the application that launched the macro, my ressources get freed. So is there a way to free my memory inside my nested loops? thanks in advance, tim -- http://mail.python.org/mailman/listinfo/python-list
Python not giving free memory back to the os get's me in real problems ...
Hi! I got same problem with Lotus Domino + Python. The lotus COM objects are not freed, so I get out from memory. I solve this problem with a little trick: I make two py applications. The first is call the second as like another application. The first is put some work to the seconds datadir. Whe second is finished with it, it is close itself. Because the memory operations are existing in only the second script, all memory get freed after it is finished, and the first (controller) py script can call the next session without any problem. dd 25 Apr 2007 07:08:42 -0700, [EMAIL PROTECTED] [EMAIL PROTECTED]: So I read quite a few things about this phenomenon in Python 2.4.x but I can hardly believe that there is really no solution to my problem. We use a commercial tool that has a macro functionality. These macros are written in python. So far nothing extraordinary. Our (python-)macro uses massively nested loops which are unfortunately necessary. These loops perform complex calculations in this commercial tool. To give you a quick overview how long this macros runs: The outer loop takes 5-7 hours for one cycle. Each cycle creates one outputfile. So we would like to perform 3-5 outer cycles en bloc. Unfortunately one of our computers (768MB RAM) crashes after just ~10% of the first cycle with the following error message: http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg while another computer (1GB RAM) crashes after ~10% of the fourth loop. While the virtual memory on the 1gb machine was full to the limit when it crashed the memory usage of the 768mb machine looked this this: http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg The moment I close the application that launched the macro, my ressources get freed. So is there a way to free my memory inside my nested loops? thanks in advance, tim -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
[EMAIL PROTECTED] wrote: So I read quite a few things about this phenomenon in Python 2.4.x but I can hardly believe that there is really no solution to my problem. We use a commercial tool that has a macro functionality. These macros are written in python. So far nothing extraordinary. Our (python-)macro uses massively nested loops which are unfortunately necessary. These loops perform complex calculations in this commercial tool. To give you a quick overview how long this macros runs: The outer loop takes 5-7 hours for one cycle. Each cycle creates one outputfile. So we would like to perform 3-5 outer cycles en bloc. Unfortunately one of our computers (768MB RAM) crashes after just ~10% of the first cycle with the following error message: http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg while another computer (1GB RAM) crashes after ~10% of the fourth loop. While the virtual memory on the 1gb machine was full to the limit when it crashed the memory usage of the 768mb machine looked this this: http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg The moment I close the application that launched the macro, my ressources get freed. So is there a way to free my memory inside my nested loops? thanks in advance, tim Let's see for this I need to get out my crystal ball... If it is a commercial application, you should contact their tech support for a solution. The problem isn't specifically a Python problem but rather an implementation problem with their app. -Larry -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
Larry Bates [EMAIL PROTECTED] wrote: Let's see for this I need to get out my crystal ball... If it is a commercial application, you should contact their tech support for a solution. The problem isn't specifically a Python problem but rather an implementation problem with their app. -Larry Well one part of the problem maybe really is on the side of this tool. Because the implementation of the macro functionality is really crappy. But unfortunately there are no competitive-product we could switch too, so we have to live with this poor environment. But I don't think that the WHOLE problem lies inside this tool. Because I read http://mail.python.org/pipermail/python-dev/2005-January/051255.html and http://evanjones.ca/python-memory.html Sadly we can't switch from Python 2.4 to 2.5 because this stupid macro functionality only supports 2.4.x I really know that all this is not the best setup to get happy with but there really is no other setup available. So I'm happy for every input all of you can give me. greetings, tim -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
On 25 Apr 2007 15:04:59 GMT, TimC [EMAIL PROTECTED] wrote: Larry Bates [EMAIL PROTECTED] wrote: Let's see for this I need to get out my crystal ball... If it is a commercial application, you should contact their tech support for a solution. The problem isn't specifically a Python problem but rather an implementation problem with their app. -Larry Well one part of the problem maybe really is on the side of this tool. Because the implementation of the macro functionality is really crappy. But unfortunately there are no competitive-product we could switch too, so we have to live with this poor environment. But I don't think that the WHOLE problem lies inside this tool. Because I read http://mail.python.org/pipermail/python-dev/2005-January/051255.html and http://evanjones.ca/python-memory.html Sadly we can't switch from Python 2.4 to 2.5 because this stupid macro functionality only supports 2.4.x I really know that all this is not the best setup to get happy with but there really is no other setup available. So I'm happy for every input all of you can give me. The problem you're seeing almost certainly isn't related to the memory arena issues referenced on the list. Returning memory to the OS doesn't affect exhaustion of your virtual address space. More likely, your nested loops are just creating more objects than is possible to hold within a single process. That may be because you're working with very large data sets, or because you have some sort of ref cycle problem, or simply because you hold onto references you no longer need. You'll need to do more work to diagnose the exact details before you can find a fix. Contacting your vendor support would be a really good first step. It's possible that you have a degenerate case that triggers some of the arena problems (like allocation 10 million integers, and then never using them again), and thats the root of your problem. Without knowing more about the usage pattern of your application, theres no way we can determine that. If you can't upgrade to python 2.5, your only solution is going to be to rework your algorithm to not do that. Again, contacting your vendor support is going to be the first step. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
On 2007-04-25, Chris Mellon [EMAIL PROTECTED] wrote: Returning memory to the OS doesn't affect exhaustion of your virtual address space. More likely, your nested loops are just creating more objects than is possible to hold within a single process. I'm a bit fuzzy on this, but I don't think there _is_ a practical way to return memory to the OS in many OSes. For example in Unix the C library uses the sbrk() call to increase the size of the data segment when additional memory is needed to handle soemthing like malloc() calls. In theory, one can use brk() to reduce the size of the data segment, but I think making the segment smaller will produce undefined behavior if any of the standard C library's dynamic memory routines (e.g. malloc/free) have ever been used by the program prior to the call to brk(). -- Grant Edwards grante Yow! FEELINGS are cascading at over me!!! visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
Grant Edwards wrote: On 2007-04-25, Chris Mellon [EMAIL PROTECTED] wrote: Returning memory to the OS doesn't affect exhaustion of your virtual address space. More likely, your nested loops are just creating more objects than is possible to hold within a single process. I'm a bit fuzzy on this, but I don't think there _is_ a practical way to return memory to the OS in many OSes. For example in Unix the C library uses the sbrk() call to increase the size of the data segment when additional memory is needed to handle soemthing like malloc() calls. In theory, one can use brk() to reduce the size of the data segment, but I think making the segment smaller will produce undefined behavior if any of the standard C library's dynamic memory routines (e.g. malloc/free) have ever been used by the program prior to the call to brk(). Interesting questions. What happens when an object is 'cleaned' up by using the 'del' command. Does the memory space stay in the python process, get handed back to the OS, or some combination of both? I remember 'C' on VMS at least, could be coerced into return memory on block boundaries. 'C++' was suppose to have garbage collect, but I was always doubtful it worked well. sph -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
[EMAIL PROTECTED] wrote: So I read quite a few things about this phenomenon in Python 2.4.x but I can hardly believe that there is really no solution to my problem. We use a commercial tool that has a macro functionality. These macros are written in python. So far nothing extraordinary. Our (python-)macro uses massively nested loops which are unfortunately necessary. These loops perform complex calculations in this commercial tool. To give you a quick overview how long this macros runs: The outer loop takes 5-7 hours for one cycle. Each cycle creates one outputfile. So we would like to perform 3-5 outer cycles en bloc. Unfortunately one of our computers (768MB RAM) crashes after just ~10% of the first cycle with the following error message: http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg while another computer (1GB RAM) crashes after ~10% of the fourth loop. While the virtual memory on the 1gb machine was full to the limit when it crashed the memory usage of the 768mb machine looked this this: http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg The moment I close the application that launched the macro, my ressources get freed. So is there a way to free my memory inside my nested loops? thanks in advance, tim Could you split the program into one handling the outer loop and calling another program, with data transfer, to handle the inner loops? - Paddy. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
On 2007-04-25, Steven Howe [EMAIL PROTECTED] wrote: I'm a bit fuzzy on this, but I don't think there _is_ a practical way to return memory to the OS in many OSes. For example in Unix the C library uses the sbrk() call to increase the size of the data segment when additional memory is needed to handle soemthing like malloc() calls. In theory, one can use brk() to reduce the size of the data segment, but I think making the segment smaller will produce undefined behavior if any of the standard C library's dynamic memory routines (e.g. malloc/free) have ever been used by the program prior to the call to brk(). Interesting questions. What happens when an object is 'cleaned' up by using the 'del' command. Does the memory space stay in the python process, get handed back to the OS, or some combination of both? Assuming the python interpreter free()s the memory, my understanding is that on Unixes the memory is returned to the pool used by malloc(), but is not returned to the OS since there isn't a practical way to ensure that the memory at the end of the data segment is not used. In theory, you could walk the data structures used by free/malloc and try to figure out if a free() should allow brk() to be called to reduce the data segment size. That would only happen if the free() call was freeing data that was at the end of the data segment. It's possible that some standard C libraries do that, but most of what I remember reading implies that they don't. I don't really keep current on libc details, so my info might be way out of date. Asking on the gnu libc mailing list would probably provide a more authoritative answer. I'd wager that they even know what other non-Gnu libc implementations do. I remember 'C' on VMS at least, could be coerced into return memory on block boundaries. You can call brk() in a C program to reduce the size of the data segment, but that may result in free and malloc breakage because you've returned (behind their back) some of the memory they are managing. 'C++' was suppose to have garbage collect, but I was always doubtful it worked well. -- Grant Edwards grante Yow! Why is everything made at of Lycra Spandex? visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
In article [EMAIL PROTECTED], Steven Howe [EMAIL PROTECTED] wrote: Interesting questions. What happens when an object is 'cleaned' up by using the 'del' command. Does the memory space stay in the python process, get handed back to the OS, or some combination of both? I remember 'C' on VMS at least, could be coerced into return memory on block boundaries. 'C++' was suppose to have garbage collect, but I was always doubtful it worked well. Note that UNIX (and VMS) use virtual memory. Real memory space gets handed back to the OS by default -- if you don't use it, you lose it. It isn't coercion, but it does happen on a per page basis, so fragmentation wastes space. If a Python program uses more space than it ought to need, then some knowledge of Python's reference counting allocation scheme will be useful. In particular, you need a fairly good grasp of what a reference is. Donn Cave, [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
Donald 'Paddy' McCarthy [EMAIL PROTECTED] wrote Could you split the program into one handling the outer loop and calling another program, with data transfer, to handle the inner loops? - Paddy. I'm afraid this isn't possible, because the python macro is called and started from within our commercial tool which leads to the cirumstance that this tool eats up more and more memory. Due to the fact that there is no way we can start the tool via commandline giving an argument that tells the tool it has to start macroXY.py after startup I don't see a way to lower the memory usage by splitting it up in two loops. thanks for all your help so far, tim -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
TimC schrieb: Donald 'Paddy' McCarthy [EMAIL PROTECTED] wrote Could you split the program into one handling the outer loop and calling another program, with data transfer, to handle the inner loops? - Paddy. I'm afraid this isn't possible, because the python macro is called and started from within our commercial tool which leads to the cirumstance that this tool eats up more and more memory. Due to the fact that there is no way we can start the tool via commandline giving an argument that tells the tool it has to start macroXY.py after startup I don't see a way to lower the memory usage by splitting it up in two loops. Is the python macro crippled in any way - e.g. threading? Maybe delegating the actual processing to an external python process you feed through the macro might work. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Python not giving free memory back to the os get's me in real problems ...
Grant Edwards wrote: Assuming the python interpreter free()s the memory, my understanding is that on Unixes the memory is returned to the pool used by malloc(), but is not returned to the OS since there isn't a practical way to ensure that the memory at the end of the data segment is not used. http://www.dent.med.uni-muenchen.de/~wmglo/malloc-slides.html Large chunks are allocated via mmap(). The threshold is set through the environment variable MALLOC_MMAP_THRESHOLD_. The maximum number of chunks allocated in this way is limited through MALLOC_MMAP_MAX_. I think these large chunks are returned to the OS when freed. Trimming occurs when the size of the top chunk exceeds MALLOC_TRIM_THRESHOLD_. I think this means that memory for small chunks is returned to the OS if the free space at the top of the heap exceeds the threshold. Asking on the gnu libc mailing list would probably provide a more authoritative answer. I'd wager that they even know what other non-Gnu libc implementations do. Agreed, they would be authoritative for GNU libc. Charles -- http://mail.python.org/mailman/listinfo/python-list