Yes, regular form parser uses mmap, so for huge files it is not good
I use 1-2GB file pretty easily with the spooler.
In the script, just check if the file is spooled
set tmpfile [ns_conn contentfile]
if { $tmpfile != "" } {
# Rename to keep it from being deleted on session exit
file rename $tmpfile $tmpfile.mpg
# Call offline parser
set form [ns_set create]
ns_parseformfile $tmpfile $form [ns_set iget [ns_conn headers] content-type]
}
Stephen Deasey wrote:
> On Thu, Nov 19, 2009 at 2:25 AM, John Buckman <j...@magnatune.com> wrote:
>> I'm writing an app that accepts large file uploads, and trying to lower the
>> memory requirements.
>>
>> When I upload large files, at the point where the files are completely
>> uploaded, Naviserver consumes an equivalent amount of memory as the file
>> uploaded size, as it hands control over to my form handler. Memory is
>> released once my form handler returns. Virtually no memory is used (thanks,
>> spooler) as the large file is being uploaded, only once the upload completes.
>>
>> I'm not sure why Naviserver needs to hold the file in memory.
>>
>> So, questions:
>>
>> a) is this by design?
>> b) has anyone on this list worked on this section of code?
>> c) any fixes, or should I plunge into the nsd/form.c code (is that right?)
>> to try to fix this?
>
>
> We originally talked about this way back in 2005:
>
> http://sourceforge.net/mailarchive/message.php?msg_id=37583D44-1F17-401D-8BCA-26F816831B92%40archiware.com
>
> Spooling got implemented later.
>
> What didn't didn't get implemented, and what you're tripping over, is
> the on-the-fly parsing of file-upload mime headers so that *only* the
> contents of the actual file becomes a file spooled on disk, and not
> the file topped and tailed with mime headers. As you've discovered,
> the way the form parsing is implemented it just reads the whole thing
> back into memory anyway, negating the benefit of spooling to disk.
>
> Ideally, the spooler thread would parse the form as it is read in
> chunks from the socket, placing variables in the form structure, and
> spooling file contents to disk. If more than one file is in the form
> then multiple spool files would be used.
>
> Watch out for encoding. IIRC the way it currently works forms can be
> reparsed if the encoding changes. Perhaps everything but the file
> parts of a multipart form could be saved in a buffer..? ie. you strip
> out the large files from the input stream and leave the rest for
> normal form processing.
>
> Also watch out for which temp directory files are being spooled to. If
> it is on a separate partition then the final rename() will actually be
> a file copy. Large files should probably be sent to a configured
> spool directory.
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
