Yes, regular form parser uses mmap, so for huge files it is not good
I use 1-2GB file pretty easily with the spooler.
In the script, just check if the file is spooled
set tmpfile [ns_conn contentfile]
if { $tmpfile != } {
# Rename to keep it from being deleted on session exit
file rename $tmpfile $tmpfile.mpg
# Call offline parser
set form [ns_set create]
ns_parseformfile $tmpfile $form [ns_set iget [ns_conn headers] content-type]
}
Stephen Deasey wrote:
On Thu, Nov 19, 2009 at 2:25 AM, John Buckman j...@magnatune.com wrote:
I'm writing an app that accepts large file uploads, and trying to lower the
memory requirements.
When I upload large files, at the point where the files are completely
uploaded, Naviserver consumes an equivalent amount of memory as the file
uploaded size, as it hands control over to my form handler. Memory is
released once my form handler returns. Virtually no memory is used (thanks,
spooler) as the large file is being uploaded, only once the upload completes.
I'm not sure why Naviserver needs to hold the file in memory.
So, questions:
a) is this by design?
b) has anyone on this list worked on this section of code?
c) any fixes, or should I plunge into the nsd/form.c code (is that right?)
to try to fix this?
We originally talked about this way back in 2005:
http://sourceforge.net/mailarchive/message.php?msg_id=37583D44-1F17-401D-8BCA-26F816831B92%40archiware.com
Spooling got implemented later.
What didn't didn't get implemented, and what you're tripping over, is
the on-the-fly parsing of file-upload mime headers so that *only* the
contents of the actual file becomes a file spooled on disk, and not
the file topped and tailed with mime headers. As you've discovered,
the way the form parsing is implemented it just reads the whole thing
back into memory anyway, negating the benefit of spooling to disk.
Ideally, the spooler thread would parse the form as it is read in
chunks from the socket, placing variables in the form structure, and
spooling file contents to disk. If more than one file is in the form
then multiple spool files would be used.
Watch out for encoding. IIRC the way it currently works forms can be
reparsed if the encoding changes. Perhaps everything but the file
parts of a multipart form could be saved in a buffer..? ie. you strip
out the large files from the input stream and leave the rest for
normal form processing.
Also watch out for which temp directory files are being spooled to. If
it is on a separate partition then the final rename() will actually be
a file copy. Large files should probably be sent to a configured
spool directory.
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
