Re: Python memory handling

2007-06-04 Thread Frédéric PICA
Greets, Sorry for my late answer, google groups lost my post... First, thanks you for your explanations about memory handling in the os and python. I've tried with python 2.5 under linux : For the parsing of a 66 Mb xml file with cElementTree : When starting python : 2.1 Mb private memory used

Re: Python memory handling

2007-06-01 Thread Leo Kislov
On May 31, 8:06 am, [EMAIL PROTECTED] wrote: Hello, I will try later with python 2.5 under linux, but as far as I can see, it's the same problem under my windows python 2.5 After reading this document :http://evanjones.ca/memoryallocator/python-memory.pdf I think it's because list or

Re: Python memory handling

2007-06-01 Thread Andrew MacIntyre
[EMAIL PROTECTED] wrote: Using the same file with cElementTree took me 217 Mb, with no unreachable object. For me it's not a good behavior, it's not a good way to let the system swap this unused memory instead of freeing it. I think it's a really good idea to have a memory pool for

Re: Python memory handling

2007-06-01 Thread Nick Craig-Wood
Andrew MacIntyre [EMAIL PROTECTED] wrote: You should also appreciate something about PyMalloc: it only handles allocation requests of 256 bytes or smaller, and this limitation is part of PyMalloc's design. If most of your allocations are 256 bytes, you're at the mercy of the platform

Python memory handling

2007-05-31 Thread frederic . pica
Greets, I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. Here is a code sample, parsing an XML file under linux python 2.4 (same problem with windows 2.5, tried with the first example) : #Python

Re: Python memory handling

2007-05-31 Thread Marc 'BlackJack' Rintsch
In [EMAIL PROTECTED], frederic.pica wrote: So as I can see, python maintain a memory pool for lists. In my first example, if I reparse the xml file, the memory doesn't grow very much (0.1 Mb precisely) So I think I'm right with the memory pool. But is there a way to force python to release

Re: Python memory handling

2007-05-31 Thread frederic . pica
On 31 mai, 14:16, Marc 'BlackJack' Rintsch [EMAIL PROTECTED] wrote: In [EMAIL PROTECTED], frederic.pica wrote: So as I can see, python maintain a memory pool for lists. In my first example, if I reparse the xml file, the memory doesn't grow very much (0.1 Mb precisely) So I think I'm

Re: Python memory handling

2007-05-31 Thread Paul Melis
Hello, [EMAIL PROTECTED] wrote: I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. [...] The same problem here with a simple file.readlines() #Python interpreter memory usage : 1.1 Mb private,

Re: Python memory handling

2007-05-31 Thread frederic . pica
On 31 mai, 16:22, Paul Melis [EMAIL PROTECTED] wrote: Hello, [EMAIL PROTECTED] wrote: I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. [...] The same problem here with a simple

Re: Python memory handling

2007-05-31 Thread Josh Bloom
If the memory usage is that important to you, you could break this out into 2 programs, one that starts the jobs when needed, the other that does the processing and then quits. As long as the python startup time isn't an issue for you. On 31 May 2007 04:40:04 -0700, [EMAIL PROTECTED] [EMAIL

Re: Python memory handling

2007-05-31 Thread frederic . pica
On 31 mai, 17:29, Josh Bloom [EMAIL PROTECTED] wrote: If the memory usage is that important to you, you could break this out into 2 programs, one that starts the jobs when needed, the other that does the processing and then quits. As long as the python startup time isn't an issue for you. On

Re: Python memory handling

2007-05-31 Thread Chris Mellon
Like: import pool pool.free() pool.limit(size in megabytes) Why not letting the user choosing that, why not giving the user more flexibility ? I will try later under linux with the latest stable python Regards, FP The idea that memory allocated to a process but not being used is a

Re: Python memory handling

2007-05-31 Thread Thorsten Kampe
* (31 May 2007 06:15:18 -0700) On 31 mai, 14:16, Marc 'BlackJack' Rintsch [EMAIL PROTECTED] wrote: In [EMAIL PROTECTED], frederic.pica wrote: And I'm not sure that the system will swap first this unused memory, it could also swap first another application... AFAIK. Definitely not, this is

Re: Python memory handling

2007-05-31 Thread Thorsten Kampe
* Chris Mellon (Thu, 31 May 2007 12:10:07 -0500) Like: import pool pool.free() pool.limit(size in megabytes) Why not letting the user choosing that, why not giving the user more flexibility ? I will try later under linux with the latest stable python Regards, FP The idea

Re: Python memory handling

2007-05-31 Thread Klaas
On May 31, 11:00 am, Thorsten Kampe [EMAIL PROTECTED] wrote: If it's swapped to disk than this is a big concern. If your Python app allocates 600 MB of RAM and does not use 550 MB after one minute and this unused memory gets into the page file then the Operating System has to allocate and

Re: Python memory handling

2007-05-31 Thread Paul Melis
[EMAIL PROTECTED] wrote: I will try later with python 2.5 under linux, but as far as I can see, it's the same problem under my windows python 2.5 After reading this document : http://evanjones.ca/memoryallocator/python-memory.pdf I think it's because list or dictionnaries are used by the

Re: Python memory handling

2007-05-31 Thread Matthew Woodcraft
Josh Bloom [EMAIL PROTECTED] wrote: If the memory usage is that important to you, you could break this out into 2 programs, one that starts the jobs when needed, the other that does the processing and then quits. As long as the python startup time isn't an issue for you. And if python startup

Re: Python memory handling

2007-05-31 Thread Chris Mellon
On 5/31/07, Thorsten Kampe [EMAIL PROTECTED] wrote: * Chris Mellon (Thu, 31 May 2007 12:10:07 -0500) Like: import pool pool.free() pool.limit(size in megabytes) Why not letting the user choosing that, why not giving the user more flexibility ? I will try later under linux