On 6/16/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > Hi! > > I ran into several problems while debuggging a problem at the > customer site related to uploading files and size of those files. > > As it looks like, the NS/AS spools all the incoming content into > the memory (urks!). Depending on the content-type, the content > may be parsed, as in case of multipar/form-data. > > This is very convenient (you need not fiddle with the low-level > stuff) but it can be extremely inefficient and bloat the memory > footprint of the server if somebody likes to upload 10's or 100's > of MB of data. > > I have examined the recent changes in the AS done by Jim which > tackle this issue. He's basically using the "maxinput" config > option to decide wether to stuff everything into memory (as usual) > or, in the case input exceeds the maxinput level, stuff the > input into a temporary file. But... the programmer is then left > alone and has to parse the incoming input by himself by using > ns_conn contentchannel command to read the content. > Even more, the form machinery which is basically parsing the > multipart data still operates on the connPtr->content which > does not reflect the entire content (the excessive is spooled > into a temp file!). > > Hm... this seems pretty unusable for this particular case to me. > > Therefore, and as usual, I have a idea/suggestion how to improve > this... > > I would take the same direction as Jim did, but will simply mmap the > temp-file to the connPtr->content! This will make all the rest > of the code work as usual. On the connection end, the file will > be unmmaped and there you go. > > One can even explore the memmap machinery and see if we can > entirely drop the temp-file and use the system paging for that: > so if the input exceeds the maxinput, we just mmap /dev/zero file > I believe something like that should be possible but will have to > double-check if this is true.
It was at one point implemented with mmap: http://cvs.sourceforge.net/viewcvs.py/aolserver/aolserver/nsd/driver.c?rev=1.34&view=markup Spooling large requests to disk is clearly necessary. Almost always, a large request needs to be stored in a file server-side anyway, so it might as well go there from the start. The only problem I see is that the calls to open a file and write content to disk are blocking, and they happen in the context of the driver thread which is busy multiplexing many non-blocking operations on sockets. Should one of the calls used to spool to disk block, everything comes to a stop. The version 3.x model was to read-ahead up to the end of headers in the driver thread, then pass the conn to a conn thread. As far as I remember, the conn thread would then spool all content to disk, but this could have been on-demand and it might have been only file-upload data. A programmer could then access the spooled data using Ns_ConnRead(), Ns_ConnReadLine() etc., or they would be called for you by ns_getform etc. The thing to note however is that all the blocking calls happened in separate conn threads. In early version 4.0 the model was changed so that the driver thread would read-ahead all data (up to Content-Length bytes) before the conn was passed to a conn thread. In a point release a limit was introduced to avoid the obvious DOS attack. This allowed an easier interface to the data for programmers: Ns_ConnContent() which returns a pointer to an array of bytes. Ns_ConnReadLine() etc. are no longer used and are currently broken. Version 4.1 work seems to be trying to tackle the problem of what happens when large files are uploaded. The version 4.0 model work extremely well for HTTP POST data up to a few K in size, but as you've noticed it really bloats memory when multi-megabyte files are uploaded. This version also introduces limits, which are a URL-specific way of pre-declaring the maxupload size and some other parameters. So anyway, here's my idea of how it should work: There's a maxreadahead parameter which is <= maxinput. When a request arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are read into a buffer by the driver thread before the conn is passed to a conn thread. The conn thread runs the registered proc for that URL. If that procedure does not try to acces the content, then when the conn is returned to the driver thread any content > maxreadahead is drained. If the registered proc does try to access the content via e.g. Ns_ConnContent() (and I think this would be the main method, used underneath by all others) and the content is <= maxreadahead, then a pointer to the readahead buffer is returned. If the content is accessed and it is > maxreadahead, a temp file is mmaped, the readahead buffer is dumped to it, and the remaining bytes are read from the socket, possibly blocking many times, and dumped to the mmaped file, again possibly blocking. A call to Ns_ConnContent() returns a pointer to the mmaped bytes of the open file. At any time before the registered proc asks for the content it can check the content length header and decide whether it is too large to accept. You could imagine setting a low global maxinput, and a call such as Ns_ConnSetMaxInput() which a registered proc could call to increase the limit for that connection only. The advantage over the limits scheme in 4.1 is that the code which checks the size of the content and processes it is kept together, rather than having to pre-decalare maxinput sizes for arbitrary URLs in the config file. This is similar to 4.1 except the task of overflowing to disk is moved to the conn thread and is lazy. The mental model is also slightly different, the assumption is that content goes to disk, but theres a cache for read-ahead which may be enough to hold everything. In 4.1 it's everything is read into memory, unless it's large in which case it overflows to disk. How does that sound?
