On 17/01/2007, at 8:54 PM, [EMAIL PROTECTED] wrote:
Thank you for your reply.
Now I use
Apache 2.2.4
Mod_python 3.3.0.b
Python 2.3.5
The results are more or less the same when I used
Apache 2.0
and mod_python 3.1.3-1
Python 2.3.4
Memory increases when I read (filter.read()) or write( filter.write
(s).
In otherwords, memory is not released after I wrote data to a file
( before next reading).
Is my presumption correct that the memory should be released
during each
reading/writing in the filter?
Or do I make a mistake somewhere?
You are giving conflicting information. Previously you said you were
running
out of resources which suggested runaway memory use. Ie., process size
just kept growing and growing until machine memory exhausted. Is that
what
is happening or were you more echoing your concerns rather than what was
happening.
What you will probably see after having restarted Apache and then
doing an
initial upload is that memory use by Apache will grow as it reads in
more and
more of the file. Once the complete request has been serviced, Apache
will
continue to use at least as much memory as it allocated by that
point. You
will not see the Apache process size shrink as once the Apache process
has allocated memory, it notionally does not give it back to the
operating system
until the process has quit. This doesn't mean it is lost though as it
is put on
a free list within the Apache process itself and when a subsequent
upload of
same file for example occurs, it will just use that free memory.
Thus, process
size will grow on the first request but if you keep doing the same
request, the
process size should stay reasonably stable.
Note that how much the Apache process grows to initially can depend a
bit
on how a handler deals with a file upload. If the response handler
does a
req.read() and then processes it all in one go, it will have to hold
the whole
file in memory and so if it is a big file it will use a lot. If
instead the response
handler read in the file in small blocks at a time using req.read
(blocksize)
and process each block and then discarded it, the memory use by Apache
will be less as it doesn't have to hold the whole file in memory but
more or
less just what it is processing.
It is hard to tell if you do have a problem with a memory leak or
whether you
just understand how memory allocation works for processes.
What are you using to monitor memory use? Does memory use keep growing
continually after every request or does it plateau after the initial
increases
with the firs requests?
Graham
Thank you for help
Regards,
Lad.
> On 16/01/2007, at 6:54 PM, [EMAIL PROTECTED] wrote:
>
> > I have this, very simple input filter
> >
> > def inputfilter(filter):
> > if filter.req.method != 'POST':
> > filter.pass_on()
> > return
> > filter.req.log_error('first read')
> > s = filter.read()
> > while s:
> > filter.req.log_error('writing (%s)' % len(s))
> > filter.write(s)
> > #filter.flush()
> > s = filter.read()
> > if s is None:
> > filter.req.log_error('closing')
> > filter.close()
> >
> > I would like to use an input filter for uploading large files.
> > I would expect that after each writing/reading, filter releases
> > memory.But instead memory
> > increases during reading/writing and no memory is released
after each
> > reading/writing. Is it a common feature that the filter during
> > reading/writing does NOT release memory and memory increases
until the
> > whole file is uploaded? Or is my presumption correct that the
memory
> > should be released during each reading/writing in the filter?
Thank you
> > for help and reply.
>
> Since I don't recollect you saying what version of mod_python you
are
> using, I would suggest you upgrade to mod_python 3.3.0b if you
aren't
> using it already. Once you have confirmed that you still have
the problem
> with that version let us know. Also state which versions of
Apache and
> Python you are using at the same time.
>
> For 3.3.0b see:
>
> http://httpd.apache.org/modules/python-download.cgi
> http://nicolas.lehuen.com/download/mod_python/
>
> The latter has Windows binaries for various Apache/Python combos.
>
> Graham
