RE: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
I think we should leave it alone. This is the difference between benchmarks and the real world. How often do people have 8 requests in a row that total less than 8K? As a compromise, there are two other options. You could have the core_output_filter refuse to buffer more than 2 requests, or you could have the core_output_filter not buffer if the full request is in the buffer. Removing the buffering is not the correct solution, because it does have a negative impact in the real world. Ryan -- Ryan Bloom [EMAIL PROTECTED] 645 Howard St. [EMAIL PROTECTED] San Francisco, CA -Original Message- From: Brian Pane [mailto:[EMAIL PROTECTED]] Sent: Sunday, June 23, 2002 9:38 PM To: [EMAIL PROTECTED] Subject: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers On Sun, 2002-06-23 at 20:58, Brian Pane wrote: For what it's worth, I just tried the test case that you posted. On my test system, 2.0 is faster when I run ab without -k, and 1.3 is faster when I run with -k. I studied this test case and found out why 2.0 runs faster in the non-keepalive case and slower in the non-keepalive case. It's because of the logic in core_output_filter() that tries to avoid small writes when c-keepalive is set. In Rasmus's test, the file size is only 1KB, so core_output_filter reads in and buffers the contents of 8 requests before it finally reaches the 8KB threshold and starts calling writev. 1.3, in contrast, writes out each request's response immediately, with no buffering. I'm somewhat inclined to remove the buffering in this case, and let core_output_filter send the response as soon as it sees EOS for each request, even if that means sending a small packet. I just tried this in my working tree, and it does speed up 2.0 for this particular test case. On the other hand, I'm not a fan of small write calls in general. Anyone else care to offer an opinion on whether we should remove the buffering in this situation? --Brian
RE: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: I think we should leave it alone. This is the difference between benchmarks and the real world. How often do people have 8 requests in a row that total less than 8K? As a compromise, there are two other options. You could have the core_output_filter refuse to buffer more than 2 requests, or you could have the core_output_filter not buffer if the full request is in the buffer. In your second option, do you mean full response rather than full request? If so, it's equivalent to what I was proposing yesterday: send the response for each request as soon as we see EOS for that request. I like that approach a lot because it keeps us from doing an extra mmap/memcpy/memunmap write before the write in the real-world case where a client sends a non-pipelined request for a small (8KB) file over a keepalive connection. I do mean full response, and that is NOT equivalent to sending the response as soon as you see the EOS for that request. EVERY request gets its own EOS. If you send the response when you see the EOS, then you will have removed all of the buffering for pipelined requests. You are trying to solve a problem that doesn't exist in the real world IIUIC. Think it through. The problem is that if the page is 1k and you request that page 8 times on the same connection, then Apache will buffer all of the responses. How often are those conditions going to happen in the real world? Now, take it one step further please. The real problem is how AB is measuring the results. If I send 3 requests, and Apache buffers the response, how is AB measuring the time? Does it start counting at the start of every request, or is the time just started at start of the first request? Perhaps a picture: Start of requestresponse received 0 5 10 15 20 25 30 35 --- request 1 --- request 2 - request 3 -- request 4 Did the 4 requests take 35 seconds total, or 85 seconds? I believe that AB is counting this as 85 seconds for 4 requests, but Apache is only taking 35 seconds for the 4 requests. Ryan
RE: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: I think we should leave it alone. This is the difference between benchmarks and the real world. How often do people have 8 requests in a row that total less than 8K? As a compromise, there are two other options. You could have the core_output_filter refuse to buffer more than 2 requests, or you could have the core_output_filter not buffer if the full request is in the buffer. In your second option, do you mean full response rather than full request? If so, it's equivalent to what I was proposing yesterday: send the response for each request as soon as we see EOS for that request. I like that approach a lot because it keeps us from doing an extra mmap/memcpy/memunmap write before the write in the real-world case where a client sends a non-pipelined request for a small (8KB) file over a keepalive connection. I do mean full response, and that is NOT equivalent to sending the response as soon as you see the EOS for that request. EVERY request gets its own EOS. If you send the response when you see the EOS, then you will have removed all of the buffering for pipelined requests. -1 on buffering across requests, because the performance problems caused by the extra mmap+munmap will offset the gain you're trying to achieve with pipelining. Wait a second. Now you want to stop buffering to fix a completely differeny bug. The idea that we can't keep a file_bucket in the brigade across requests is only partially true. Take a closer look at what we are doing and why when we convert the file to an mmap. That logic was added so that we do not leak file descriptors across requests. However, there are multiple options to fix that. The first, and easiest one is to just have the core_output_filter call apr_file_close() after it has sent the file. The second is to migrate the apr_file_t to the conn_rec's pool if and only if the file needs to survive the request's pool being killed. Because we are only migrating file descriptors in the edge case, this shouldn't cause a big enough leak to cause a problem. You are trying to solve a problem that doesn't exist in the real world IIUIC. Think it through. The problem is that if the page is 1k and you request that page 8 times on the same connection, then Apache will buffer all of the responses. How often are those conditions going to happen in the real world? That's not the problem that I care about. The problem that matters is the one that happens in the real world, as a side-effect of the core_output_filter() code trying to be too clever: - Client opens a keepalive connection to httpd - Cclient requests a file smaller than 8KB - core_output_filter(), upon seeing the file bucket followed by EOS, decides to buffer the output because it has less than 8KB total. - There isn't another request ready to be read (read returns EAGAIN) because the client isn't pipelining its connections. So we then do the writev of the file that we've just finished buffering. But, this case only happens if the headers + the file are less than 8k. If the file is 10k, then this problem doesn't actually exist at all. As I said above, there are better ways to fix this than removing all ability to pipeline responses. Aside from slowing things down for the user, this hurts the scalability of the httpd (mmap and munmap don't scale well on multiprocessor boxes). What we should be doing in this case is just doing the write immediately upon seeing the EOS, rather than penalizing both the client and the server. By doing that, you are removing ANY benefit to using pipelined requests when serving files. Multiple research projects have all found that pipelined requests show a performance benefit. In other words, you are fixing a performance problem by removing another performance enhancer. Ryan
Re: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: I think we should leave it alone. This is the difference between benchmarks and the real world. How often do people have 8 requests in a row that total less than 8K? As a compromise, there are two other options. You could have the core_output_filter refuse to buffer more than 2 requests, or you could have the core_output_filter not buffer if the full request is in the buffer. In your second option, do you mean full response rather than full request? If so, it's equivalent to what I was proposing yesterday: send the response for each request as soon as we see EOS for that request. I like that approach a lot because it keeps us from doing an extra mmap/memcpy/memunmap write before the write in the real-world case where a client sends a non-pipelined request for a small (8KB) file over a keepalive connection. I do mean full response, and that is NOT equivalent to sending the response as soon as you see the EOS for that request. EVERY request gets its own EOS. If you send the response when you see the EOS, then you will have removed all of the buffering for pipelined requests. -1 on buffering across requests, because the performance problems caused by the extra mmap+munmap will offset the gain you're trying to achieve with pipelining. Wait a second. Now you want to stop buffering to fix a completely differeny bug. The idea that we can't keep a file_bucket in the brigade across requests is only partially true. Take a closer look at what we are doing and why when we convert the file to an mmap. That logic was added so that we do not leak file descriptors across requests. However, there are multiple options to fix that. The first, and easiest one is to just have the core_output_filter call apr_file_close() after it has sent the file. The second is to migrate the apr_file_t to the conn_rec's pool if and only if the file needs to survive the request's pool being killed. Because we are only migrating file descriptors in the edge case, this shouldn't cause a big enough leak to cause a problem. You are trying to solve a problem that doesn't exist in the real world IIUIC. Think it through. The problem is that if the page is 1k and you request that page 8 times on the same connection, then Apache will buffer all of the responses. How often are those conditions going to happen in the real world? That's not the problem that I care about. The problem that matters is the one that happens in the real world, as a side-effect of the core_output_filter() code trying to be too clever: - Client opens a keepalive connection to httpd - Cclient requests a file smaller than 8KB - core_output_filter(), upon seeing the file bucket followed by EOS, decides to buffer the output because it has less than 8KB total. - There isn't another request ready to be read (read returns EAGAIN) because the client isn't pipelining its connections. So we then do the writev of the file that we've just finished buffering. But, this case only happens if the headers + the file are less than 8k. If the file is 10k, then this problem doesn't actually exist at all. As I said above, there are better ways to fix this than removing all ability to pipeline responses. Aside from slowing things down for the user, this hurts the scalability of the httpd (mmap and munmap don't scale well on multiprocessor boxes). What we should be doing in this case is just doing the write immediately upon seeing the EOS, rather than penalizing both the client and the server. By doing that, you are removing ANY benefit to using pipelined requests when serving files. Multiple research projects have all found that pipelined requests show a performance benefit. In other words, you are fixing a performance problem by removing another performance enhancer. Ryan Ryan, Solve the problem to enable setting aside the open fd just long enough to check for a pipelined request will nearly completely solve the worst part (the mmap/munmap) of this problem. On systems with expensive syscalls, we can do browser detection and dynamically determine whether we should attempt the pipelined optimization or not. Not many browsers today support pipelining requests, FWIW. Bill
RE: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
From: Bill Stoddard [mailto:[EMAIL PROTECTED]] From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: From: Brian Pane [mailto:[EMAIL PROTECTED]] Ryan Bloom wrote: Wait a second. Now you want to stop buffering to fix a completely differeny bug. The idea that we can't keep a file_bucket in the brigade across requests is only partially true. Take a closer look at what we are doing and why when we convert the file to an mmap. That logic was added so that we do not leak file descriptors across requests. However, there are multiple options to fix that. The first, and easiest one is to just have the core_output_filter call apr_file_close() after it has sent the file. The second is to migrate the apr_file_t to the conn_rec's pool if and only if the file needs to survive the request's pool being killed. Because we are only migrating file descriptors in the edge case, this shouldn't cause a big enough leak to cause a problem. You are trying to solve a problem that doesn't exist in the real world IIUIC. Think it through. The problem is that if the page is 1k and you request that page 8 times on the same connection, then Apache will buffer all of the responses. How often are those conditions going to happen in the real world? That's not the problem that I care about. The problem that matters is the one that happens in the real world, as a side-effect of the core_output_filter() code trying to be too clever: - Client opens a keepalive connection to httpd - Cclient requests a file smaller than 8KB - core_output_filter(), upon seeing the file bucket followed by EOS, decides to buffer the output because it has less than 8KB total. - There isn't another request ready to be read (read returns EAGAIN) because the client isn't pipelining its connections. So we then do the writev of the file that we've just finished buffering. But, this case only happens if the headers + the file are less than 8k. If the file is 10k, then this problem doesn't actually exist at all. As I said above, there are better ways to fix this than removing all ability to pipeline responses. Aside from slowing things down for the user, this hurts the scalability of the httpd (mmap and munmap don't scale well on multiprocessor boxes). What we should be doing in this case is just doing the write immediately upon seeing the EOS, rather than penalizing both the client and the server. By doing that, you are removing ANY benefit to using pipelined requests when serving files. Multiple research projects have all found that pipelined requests show a performance benefit. In other words, you are fixing a performance problem by removing another performance enhancer. Ryan Ryan, Solve the problem to enable setting aside the open fd just long enough to check for a pipelined request will nearly completely solve the worst part (the mmap/munmap) of this problem. On systems with expensive syscalls, we can do browser detection and dynamically determine whether we should attempt the pipelined optimization or not. Not many browsers today support pipelining requests, FWIW. That would be a trivial change. I'll have a patch posted for testing later today. Ryan
RE: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
-1 on buffering across requests, because the performance problems caused by the extra mmap+munmap will offset the gain you're trying to achieve with pipelining. Wait a second. Now you want to stop buffering to fix a completely differeny bug. The idea that we can't keep a file_bucket in the brigade across requests is only partially true. Take a closer look at what we are doing and why when we convert the file to an mmap. That logic was added so that we do not leak file descriptors across requests. However, there are multiple options to fix that. The first, and easiest one is to just have the core_output_filter call apr_file_close() after it has sent the file. The second is to migrate the apr_file_t to the conn_rec's pool if and only if the file needs to survive the request's pool being killed. Because we are only migrating file descriptors in the edge case, this shouldn't cause a big enough leak to cause a problem. Migrating the apr_file_t to the conn_rec's pool is an appealing solution, but it's quite dangerous. With that logic added to the current 8KB threshold, it would be too easy to make an httpd run out of file descriptors: Send a pipeline of a few hundred requests for some tiny file (e.g., a small image). They all get setaside into the conn_rec's pool. Then send a request for something that takes a long time to process, like a CGI. Run multiple copies of this against the target httpd at once, and you'll be able to exhaust the file descriptors for a threaded httpd all too easily. That is why we allow people to control how many requests can be sent on the same connection. Or, you can just have a limit on the number of file descriptors that you are willing to buffer. And, the pipe_read function should be smart enough that if we don't get any data off of the pipe, for say 30 seconds, then we flush whatever data we currently have. That's not the problem that I care about. The problem that matters is the one that happens in the real world, as a side-effect of the core_output_filter() code trying to be too clever: - Client opens a keepalive connection to httpd - Cclient requests a file smaller than 8KB - core_output_filter(), upon seeing the file bucket followed by EOS, decides to buffer the output because it has less than 8KB total. - There isn't another request ready to be read (read returns EAGAIN) because the client isn't pipelining its connections. So we then do the writev of the file that we've just finished buffering. But, this case only happens if the headers + the file are less than 8k. If the file is 10k, then this problem doesn't actually exist at all. As I said above, there are better ways to fix this than removing all ability to pipeline responses. We're not removing the ability to pipeline responses. You are removing a perfectly valid optimization to stop us from sending a lot of small packets across pipelined responses. Ryan
Re: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
On Mon, Jun 24, 2002 at 01:07:48AM -0400, Cliff Woolley wrote: Anyway, what I'm saying is: don't make design decisions of this type based only on the results of an ab run. +1 I think at this point ab should have the ability to interleave issuing new connections, handling current requests, and closing finished requests, but it would be nice if someone else could make sure. If I get a chance I'll try to run flood against the two scenarios -- it tends to get around some of the concurrency problems we see with ab but at the expense of scalability. -aaron
Re: core_output_filter buffering for keepalives? Re: Apache 2.0 Numbers
Bill Stoddard wrote: . So changing the AP_MIN_BYTES_TO_WRITE just moves the relative postion of the write() and the check pipeline read. It has one other side-effect, though, and that's what's bothering me: In the case where core_output_filter() decides to buffer a response because it's smaller than 8KB, the end result is to turn: sendfile into: mmap memcpy munmap ... buffer some more requests' output until we have 8KB ... writev ... Yack... just noticed this too. This renders the fd cache (in mod_mem_cache) virtually useless. Not sure why we cannot setaside a fd. Bill
