On 6/16/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 16.06.2005 um 23:35 schrieb Vlad Seryakov: > > > I understand that but my question is: > > How and when all content will be downloaded? > > By the driver? By the conn thread on first call? > > > > The driver will load up to maxreadahead. > > At the first call to Ns_ConnContent, the rest > will be sinked from the socket into the buffer > (until it fills) and the rest will be sinked > into the mmaped temp file. This is now done > in the connection thread. > > > Currently driver reads everything into memory. > > > > Yes, which is a problem. So it will be split between > the driver thread (up to maxreadahead) and connection > thread (the rest, if available).
Yes, exactly right. You could however set maxreadahead == maxinput for exactly the same behaviour as you get now, all content will be read by the driver thread before a conn thread ever gets woken. Obviously this is somewhat of a juggling act. What's the largest request body size you ever expect, what's the frequency of these large requests compared to small requests, therefore how big should the readahead buffer be to maximize performance but minimize memory bloat? The other resource being used is the conn thread itself, which is expensive because of the attached Tcl interp and all its procs in both text and byte code form. For every request body > maxreadahead an expensive conn thread will have to block waiting for data over the network, causing a context switch. One option here would be a pool of lightweight threads whose only job is to read data from the network and write it to disk. Once the readahead buffer is full, the connection would be passed to a lightweight thread which would spool all the data to disk, and then pass the connection to a conn thread. I don't think it's worth the complexity, not at this stage. We already have a more general purpose partitioning scheme, connection thread pools. In this context, you assume that only few URLs need to accept large request bodies and so you partition them into their own thread pool. Now, should a group of poor modem users start dribbling 20MB of data1k at a time, you may run out of photo upload threads, but the rest of your service should be fine. Hope I didn't confuse the issue here, but Vlad seemed worried about having connection threads tied up handling blocking IO. We have some good options: adjust the readahead buffer, upto maxinput if needed, and use connection thread pools to partition resource intensive URLs.
