On 6/17/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 17.06.2005 um 09:06 schrieb Stephen Deasey: > > > > However... Vlad's question about ns_getform has me wondering about > > the common case of HTTP file upload. Consider a 10MB file upload -- > > just before the form parsing code does it's thing, the situation is > > that there is a file on disk containing the 10MB of data, plus the > > mime header, plus the content separator as a trailer. What you want > > is a file with just the 10MB of data. You can truncate a file to trim > > off the trailer, but you can't trim the mime header at the beginning. > > The 10MB of data will have to be copied to a new file, and the spool > > file will be deleted when the connection is closed. > > > > This is no worse than the situation we're currently in or that of the > > 4.1 code base. I'm not sure what 3.x did here. But it does seem a bit > > unfortunate. > > > > How about this: we introduce a new private interface something like > > NsConnPeek(Ns_Conn *conn, int *lenPtr) which returns a pointer into > > the read-ahead buffer just after the HTTP headers, *without* first > > spooling the entire request body to disk. The form processing code > > would use this to hopefully parse the mime header, and probably some > > standard www-form-urlencoded data that was sent at the same time. It > > would then reset the current offset into the buffer (which was > > previously positioned at the end of the HTTP headers) so that when the > > spooling code takes over it begins at the start of the file content. > > The spool file would then be truncated to remove the trailing content > > separator. > > This assumes that form elements are packed before the actual data > in the multipart container. But what if not? Then you'd still need > to get the whole thing into the tempfile. > I beleive that's why you said "hopefully" above, right?
Hopefully in that all the data you need to pre-parse fits in the read-ahead buffer. It's not a problem if the extra form data comes after the file upload data. Files can be truncated, but they can't have data trimmed from the beginning. That's what this is all about: we're setting the read offset so as to 'trim' the beginning of the spool file contents. It can later be truncated however you want. Truncating is cheap, trimming is expensive because it means copying data. > > > > For multiple simultaneous file uploads you'd have to revert to copy > > the data to new files. But for the common case, we'd have something > > pretty efficient without too much complexity and retaining the nice > > interface that mmap allows. > > > > I understand. I believe we can add this as one of the improvements > after the first (bigger) change is done. Yeah, looks like a mult-stage project. > What I also thought about is: a way of somehow register some Tcl > callbacks when accepting a large input. If a guy uploads 10MB file > over slow connection, the browser just does not give you any clue > how for it went with uplöading. OTOH, on the server we know this. > We could register a Tcl callback to monitor the upload process, > store this data in shared memory so people could write a poll-page > which just examines this statistics-data and gives the user (over > another > connection) the idea how much has been received. > Actually, we implemented this in 3.x server and it worked fine > (since the whole upload thing was actually done in Tcl alone). > In 4.x it stopped working because of the fact that all receiving > and parsing is done internally. Hmm, not sure what the best way to do that is. You can of course fake it: as soon as the connection which accepts the large file runs it begins a server push, the first page of which contains an animated gif 'uploading...'. Once the upload completes the server pushes a new page which confirms the upload. You don't get actual upload progress, but it does reassure users that something is happening and prevents them from pressing the upload button again...
